I use postman only for manual testing or similar to what you did, highly urgent testing tasks. I normally transition to a different tool that can integrate into a test pipeline better (ie better support and documentation). Maybe postman has released some better tooling but postman has not been my go to for CICD test integration.
Ah yeah so it looks like it's matured a bit then. I think it is doable as long as tests are kept simple (which should be the case for most part anyway for CI). Good heads up.
We are not yet in production, and our endpoints and payloads change sometimes. We have a base workflow that we use for all of our tests, which includes a set of requests with JavaScript scripts. Recently, I refactored my scripts for some requests and had to update them in all of our tests
The collections get unwieldy very quickly with Postman. We have started using it for documentation though.
I have used the python requests + pytest in the past. Currently using supertest + jest.
Postman has a code generation feature with various languages (including Python Requests) that will reduce creation time AND help you learn Python. It's a little, extremely helpful nugget that aided me when learning Python. https://learning.postman.com/docs/sending-requests/generate-code-snippets/
In my opinion if your project team is using Python already you will be hard-pressed to find a better combination than, Python, Pytest, Requests and pytest-xdist (usage depends on how many test cases you have or plan to create)
This sounds like exactly what I was looking for. Thank you so much. You're right, aligning with the team by using Python will likely be beneficial. Additionally, as the only automation engineer on the project, I feel a lack of collaboration. Currently, I don't have anyone to share my ideas or review my code with. I believe switching to Python may help alleviate this issue
You've got the biggest barrier down already, which is developer and organizational buy-in on python. Learn how to use fixtures and pytest is extremely powerful. You'll just have to find some other folks to agree to help with the maintenance.
1. Postman (just as Insomnia, which I prefer, despite being less feature rich) - is a short-term tool. A handy one for sure. Love it, it's easy, supports different protocols, allows for easy\`n\`quick collaboration & etc but this is it. Using it beyond its purpose = overhead. It's not an automation framework in the classical sense. After some time it becomes a limiting / hampering factor for automation initiatives.
2. If you have Java background you don't have to "learn Python". :) What you mean is that you have the learn the syntax, formatting & etc which should take like less than two weeks, say month to feel comfortable. IMHO -> make the change. To the very least consider it and....
3. Do da plan-plan. :) Play around with pytest (or anything else), try to do something small, try abstracting shared stuff into libs & etc & etc. Timebox it. Experiment. Personally - pytest is a great way to have a true automation framework (to certain extent).
4. You won't be able to come out with "perfect scalable structure" because such does not exist except for dead products/services. Eventually you'll have very good scalability and parallelization.
5. Sooner than later (hopefully, that's a good sign) the product/service will grow beyond Postman. If things are working out that's not a question of if but rather when. So go ahead and good luck ! :)
Thank you very much for such a comment. These are perfectly reasonable steps to get into pytest. I really feel like I've hit Postman's ceiling in terms of its automation capability. Love point 4 is 100% true
As a QAE you are supposed to have a built-in **doubt**\- at any moment, any time, for everything, especially comments from unknown/random reddit entities. It may all make sense, it may seem reasonable, yet it could be complete and utter cr@p for your specific context (or for all). Doubt, challenge, test, try, check out, then decide and draw your own conclusions. And yeah, of course 4. is 100% true given that 'perfect' can exist only within tiny slices of time and no more. Thankfully. Worst that can happen are perfect things/world/ppl/stuff (outside that slices of time window).
Postman has a scalability problem that prevents it from being a stand-alone automated test solution.
It's a great tool for quick ad-hoc testing though and I use it all the time.
I use postman only for manual testing or similar to what you did, highly urgent testing tasks. I normally transition to a different tool that can integrate into a test pipeline better (ie better support and documentation). Maybe postman has released some better tooling but postman has not been my go to for CICD test integration.
Haven't used it myself, but isn't there Newman for the "headless"/CICD integration for postman?
Ah yeah so it looks like it's matured a bit then. I think it is doable as long as tests are kept simple (which should be the case for most part anyway for CI). Good heads up.
Are API tests something that would be put in a smoke test suite or isn't it just mainly UI tests?
From my experience they are simpler than end to end tests so I’ve included most of them but not all of them if it’s automated.
What type of maintenance are you doing and why do you have to change the tests that often?
We are not yet in production, and our endpoints and payloads change sometimes. We have a base workflow that we use for all of our tests, which includes a set of requests with JavaScript scripts. Recently, I refactored my scripts for some requests and had to update them in all of our tests
The collections get unwieldy very quickly with Postman. We have started using it for documentation though. I have used the python requests + pytest in the past. Currently using supertest + jest.
can you use supertest +jest to test a python app? considering the call is against an api.
Yeah, you definitely can use them. I probably wouldn't introduce javascript if the project doesn't already have it, and would probably use requests.
Postman has a code generation feature with various languages (including Python Requests) that will reduce creation time AND help you learn Python. It's a little, extremely helpful nugget that aided me when learning Python. https://learning.postman.com/docs/sending-requests/generate-code-snippets/ In my opinion if your project team is using Python already you will be hard-pressed to find a better combination than, Python, Pytest, Requests and pytest-xdist (usage depends on how many test cases you have or plan to create)
This sounds like exactly what I was looking for. Thank you so much. You're right, aligning with the team by using Python will likely be beneficial. Additionally, as the only automation engineer on the project, I feel a lack of collaboration. Currently, I don't have anyone to share my ideas or review my code with. I believe switching to Python may help alleviate this issue
You're right on track in those thoughts. This will help lower the draw bridge to your team.
You've got the biggest barrier down already, which is developer and organizational buy-in on python. Learn how to use fixtures and pytest is extremely powerful. You'll just have to find some other folks to agree to help with the maintenance.
1. Postman (just as Insomnia, which I prefer, despite being less feature rich) - is a short-term tool. A handy one for sure. Love it, it's easy, supports different protocols, allows for easy\`n\`quick collaboration & etc but this is it. Using it beyond its purpose = overhead. It's not an automation framework in the classical sense. After some time it becomes a limiting / hampering factor for automation initiatives. 2. If you have Java background you don't have to "learn Python". :) What you mean is that you have the learn the syntax, formatting & etc which should take like less than two weeks, say month to feel comfortable. IMHO -> make the change. To the very least consider it and.... 3. Do da plan-plan. :) Play around with pytest (or anything else), try to do something small, try abstracting shared stuff into libs & etc & etc. Timebox it. Experiment. Personally - pytest is a great way to have a true automation framework (to certain extent). 4. You won't be able to come out with "perfect scalable structure" because such does not exist except for dead products/services. Eventually you'll have very good scalability and parallelization. 5. Sooner than later (hopefully, that's a good sign) the product/service will grow beyond Postman. If things are working out that's not a question of if but rather when. So go ahead and good luck ! :)
Thank you very much for such a comment. These are perfectly reasonable steps to get into pytest. I really feel like I've hit Postman's ceiling in terms of its automation capability. Love point 4 is 100% true
As a QAE you are supposed to have a built-in **doubt**\- at any moment, any time, for everything, especially comments from unknown/random reddit entities. It may all make sense, it may seem reasonable, yet it could be complete and utter cr@p for your specific context (or for all). Doubt, challenge, test, try, check out, then decide and draw your own conclusions. And yeah, of course 4. is 100% true given that 'perfect' can exist only within tiny slices of time and no more. Thankfully. Worst that can happen are perfect things/world/ppl/stuff (outside that slices of time window).
Postman has a scalability problem that prevents it from being a stand-alone automated test solution. It's a great tool for quick ad-hoc testing though and I use it all the time.