This is part of the Semicolon&Sons Code Diary - consisting of lessons learned on the job. You're in the encoding category.
Last Updated: 2024-10-12
When creating an endpoint for receiving Amazon SNS notifications, I received JSON like the following:
"{\"Type\":\"Notification\",\"MessageId\":\"afa70128-12cf-5285-980c-54353313a8e0\",\"TopicArn\":\"arn:aws:sns:us-east-1:734356286499:less-penguiny-email-bounce\",\"Message\":\"{\\\"notificationType\\\":\\\"Bounce\\\",\\\"bounce\\\":{\\\"bounceType\\\":\\\"Permanent\\\",\\\"bounceSubType\\\":\\\"General\\\",\\\"bouncedRecipients\\\":[{\\\"emailAddress\\\":\\\"helloasdf@asd.cmop\\\",\\\"action\\\":\\\"failed\\\",\\\"status\\\":\\\"5.4.4\\\",\\\"diagnosticCode\\\":\\\"smtp;
550 5.4.4 Invalid
domain\\\"}],\\\"timestamp\\\":\\\"2019-04-02T08:38:52.330Z\\\",\\\"feedbackId\\\":\\\"01000169dd337a6c-59c15c30-a62c-4897-a1dd-4962dbaab1e8-000000\\\",\\\"reportingMTA\\\":\\\"dsn;
a8-27.smtp-out.amazonses.com\\\"},\\\"mail\\\":{\\\"timestamp\\\":\\\"2019-04-02T08:38:51.000Z\\\",\\\"source\\\":\\\"<no-reply@redcated.com>\\\",\\\"sourceArn\\\":\\\"arn:aws:ses:us-east-1:734356286499:identity/no-reply@redcated.com\\\",\\\"sourceIp\\\":\\\"3.81.81.236\\\",\\\"sendingAccountId\\\":\\\"734356286499\\\",\\\"messageId\\\":\\\"01000169dd3378bc-f8b21149-3159-4215-85a9-ba187f5e3cec-000000\\\",\\\"destination\\\":[\\\"helloasdf@asd.cmop\\\"]}}\",\"Timestamp\":\"2019-04-02T08:38:52.372Z\",\"SignatureVersion\":\"1\",\"Signature\":\"DNa7xn0q9fTqR0V+ubNEfWySYhIEIjF/GwrH4vfjw6Ze77zOXTJA8fzBZugxNF2+j+XAjcfkvFr0p3IK7/POMm9mF04KVujGkFVrYjYBrUELWZsB+7TCvAru7uOHmKwWh4usZVJKq28DCVy0awufHTdo6/8ts6Y4/K8/Q9ua6ATbV8Cnw9nYhJnNAWmY8Zji4OCg3NemnjMiB0PkOJjNljE3vEiTV/FtL6yIjoCVK4rHbevPwl2t+Exg86oyEQlqIvyRIRKE8OSElIDVN6qpZcFxpuzWayfc17WpIb4RmotYGuC3ryt7EX/2H1aFvLWxBumVw8aN3iEibjido9/Miw==\",\"SigningCertURL\":\"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-6aad65c2f9911b05cd53efda11f913f9.pem\",\"UnsubscribeURL\":\"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:734356286499:less-penguiny-email-bounce:3ff961a9-aa59-416a-93bc-7bfc67f91229\"}"
Running JSON.parse
gave me:
{"Type"=>"Notification", "MessageId"=>"afa70128-12cf-5285-980c-54353313a8e0", "TopicArn"=>"arn:aws:sns:us-east-1:734356286499:less-penguiny-email-bounce", "Message"=>"{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"helloasdf@asd.cmop\",\"action\":\"failed\",\"status\":\"5.4.4\",\"diagnosticCode\":\"smtp; 550 5.4.4 Invalid domain\"}],\"timestamp\":\"2019-04-02T08:38:52.330Z\",\"feedbackId\":\"01000169dd337a6c-59c15c30-a62c-4897-a1dd-4962dbaab1e8-000000\",\"reportingMTA\":\"dsn; a8-27.smtp-out.amazonses.com\"},\"mail\":{\"timestamp\":\"2019-04-02T08:38:51.000Z\",\"source\":\"Less Penguiny <no-reply@redcated.com>\",\"sourceArn\":\"arn:aws:ses:us-east-1:734356286499:identity/no-reply@redcated.com\",\"sourceIp\":\"3.81.81.236\",\"sendingAccountId\":\"734356286499\",\"messageId\":\"01000169dd3378bc-f8b21149-3159-4215-85a9-ba187f5e3cec-000000\",\"destination\":[\"helloasdf@asd.cmop\"]}}", "Timestamp"=>"2019-04-02T08:38:52.372Z", "SignatureVersion"=>"1", "Signature"=>"DNa7xn0q9fTqR0V+ubNEfWySYhIEIjF/GwrH4vfjw6Ze77zOXTJA8fzBZugxNF2+j+XAjcfkvFr0p3IK7/POMm9mF04KVujGkFVrYjYBrUELWZsB+7TCvAru7uOHmKwWh4usZVJKq28DCVy0awufHTdo6/8ts6Y4/K8/Q9ua6ATbV8Cnw9nYhJnNAWmY8Zji4OCg3NemnjMiB0PkOJjNljE3vEiTV/FtL6yIjoCVK4rHbevPwl2t+Exg86oyEQlqIvyRIRKE8OSElIDVN6qpZcFxpuzWayfc17WpIb4RmotYGuC3ryt7EX/2H1aFvLWxBumVw8aN3iEibjido9/Miw==", "SigningCertURL"=>"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-6aad65c2f9911b05cd53efda11f913f9.pem", "UnsubscribeURL"=>"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:734356286499:less-penguiny-email-bounce:3ff961a9-aa59-416a-93bc-7bfc67f91229"}
=> {"Type"=>"Notification",
"MessageId"=>"afa70128-12cf-5285-980c-54353313a8e0",
"Message"=>
"{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceType\":\"Permanent\",\"bounceSubType\":\"General\",\"bouncedRecipients\":[{\"emailAddress\":\"helloasdf@asd.cmop\",\"action\":\"failed\",\"status\":\"5.4.4\",\"diagnosticCode\":\"smtp; 550 5.4.4 Invalid domain\"}],\"timestamp\":\"2019-04-02T08:38:52.330Z\",\"feedbackId\":\"01000169dd337a6c-59c15c30-a62c-4897-a1dd-4962dbaab1e8-000000\",\"reportingMTA\":\"dsn; a8-27.smtp-out.amazonses.com\"},\"mail\":{\"timestamp\":\"2019-04-02T08:38:51.000Z\",\"source\":\"",\"sourceArn\":\"arn:aws:ses:us-east-1:734356286499:identity/no-reply@redacted.com\",\"sourceIp\":\"3.81.81.236\",\"sendingAccountId\":\"734356286499\",\"messageId\":\"01000169dd3378bc-f8b21149-3159-4215-85a9-ba187f5e3cec-000000\",\"destination\":[\"helloasdf@asd.cmop\"]}}",
"Timestamp"=>"2019-04-02T08:38:52.372Z",
"SignatureVersion"=>"1",
"Signature"=>
"DNa7xn0q9fTqR0V+ubNEfWySYhIEIjF/GwrH4vfjw6Ze77zOXTJA8fzBZugxNF2+j+XAjcfkvFr0p3IK7/POMm9mF04KVujGkFVrYjYBrUELWZsB+7TCvAru7uOHmKwWh4usZVJKq28DCVy0awufHTdo6/8ts6Y4/K8/Q9ua6ATbV8Cnw9nYhJnNAWmY8Zji4OCg3NemnjMiB0PkOJjNljE3vEiTV/FtL6yIjoCVK4rHbevPwl2t+Exg86oyEQlqIvyRIRKE8OSElIDVN6qpZcFxpuzWayfc17WpIb4RmotYGuC3ryt7EX/2H1aFvLWxBumVw8aN3iEibjido9/Miw==",
"SigningCertURL"=>"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-6aad65c2f9911b05cd53efda11f913f9.pem",
"UnsubscribeURL"=>"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:734356286499:decated-email-bounce:3ff961a9-aa59-416a-93bc-7bfc67f91229"}
I then fed this into my own functions and had surprising failures about the 'bouncedRecipients' property being called on nil.
What was up? Had I looked more closely at the parsed JSON above, I would have seen that the "Message" key contained further parsed JSON that I would have to parse again:
JSON.parse(previous_json["Message"])
=> {"notificationType"=>"Bounce",
"bounce"=>
{"bounceType"=>"Permanent",
"bounceSubType"=>"General",
"bouncedRecipients"=>[{"emailAddress"=>"wanttoseenewposts@butdonthabituallyusemailinglists.ever", "action"=>"failed", "status"=>"5.4.4", "diagnosticCode"=>"smtp; 550 5.4.4 Invalid domain"}],
"timestamp"=>"2019-04-02T08:39:05.837Z",
"feedbackId"=>"01000169dd33af2a-a381083c-283c-4b07-af90-534c33ee1af6-000000",
"reportingMTA"=>"dsn; a8-24.smtp-out.amazonses.com"},
"mail"=>
{"timestamp"=>"2019-04-02T08:39:05.000Z",
"sourceIp"=>"3.81.81.236",
"sendingAccountId"=>"734356286499",
"messageId"=>"01000169dd33ae8d-36e36e3e-af56-4660-a431-3a4013f38f92-000000",
"destination"=>["wanttoseenewposts@butdonthabituallyusemailinglists.ever"]}}
Be aware that some programs can give you serialized JSON containing sub-keys that are encoded once again in serialized JSON. Therefore you'll have to decode it twice to extract the information.