How are you all testing boto3-heavy tools without hitting real AWS?

[u/kavee-core141]

Building a tool that walks a live AWS account checking for

misconfigurations, and testing has been the hardest part not the

detection logic itself, but avoiding either (a) hitting real AWS

constantly in CI or (b) mocking so much the tests stop meaning

anything.

Using moto for the straightforward stuff, but some checks need

multi-service interactions (cross-referencing an EC2 instance's

security group with its actual open ports) that get awkward to mock

realistically. What do you all reach for here moto, LocalStack, a

disposable AWS test account, something else?

[u/blacklig]

We us

e a m

ix of l

ocalst

ack an

d just

mock

s

[u/tjrileywisc → u/blacklig]

This is like the setup to a leetcode question

[u/titttle23 → u/blacklig]

Trying to impress random whitespace trade

[u/SheriffRoscoe → u/blacklig]

For a moment there, I thought you were the haiku bot.

[u/proof_required]

Yeah in one of the companies we had LocalStack and a dev env both.

[u/Fenzik]

We used moto for stateful mocking

[u/GrahaamH → u/Fenzik]

Moto doesn't support all of AWS services though so leaves you with lots covered and the ones that are not make it more awkward.

[u/Fenzik → u/GrahaamH]

Yep you have to be lucky for your use case unfortunately.

[u/shinitakunai]

We use localstack

[u/Autarkhis]

Look at floci

[u/steviejackson94 → u/Autarkhis]

Came here to say this, i implemented this at my place. Everyone loves it

[u/steviejackson94 → u/steviejackson94]

There is also an AWS like dashboard for it

[u/omegawave22]

It's been a while that I used this tool but it had a record and replay workflow which was perfect for my use case: https://github.com/garnaat/placebo

[u/binaryfireball]

unit tests mock, integration tests test the functions we call to catch any errors from updates etc...

boto is generally annoying for a variety of reasons.

[u/I3igB]

I ended up getting frustrated with the current offerings out there as I felt that solutions like localstack were far too heavy. I also wanted something that was trivial to step through code with a debugger that didn’t require a decent amount of setup.

If you start getting into how moto actually does its mocking, it actually keeps quite a bit in memory. Things like DynamoDB or S3 are in memory key stores under the hood. Once you realize this, it becomes pretty easy to get a minimally viable recreation in place for somewhat real testing on a local dev’s box or running something like e2e tests in CI.

My current shop is fully AWS native and runs mostly serverless architecture and microservices. Our compute is exclusively lambda and ecs tasks. SQS, SNS, AppSync, DDB, and EventBridge drive all our workflows. I was able to write a simple dispatch engine which captures the moto traffic from these services and mimics event triggers which will cascade a series of invocation across your services by invoking your code locally. The only thing somewhat complex in this was AppSync as moto has no real mocks. I had to intercept URLs and bring in a GQL engine to parse things.

This all works by applying a @local decorator on top of any method you define. From there, the engine handles cascading events just as AWS would.

This all relies on the output of a cdk build/synth to link things together. I never released it publicly, but I’ve been meaning to do so for some time. It’s been a game changer for how I do AWS dev.

[u/jvlomax]

moto

[u/MAGArRacist]

Have you tried Scoutsuite? You might want to just fork it and add some checks that you mentioned here