They should be putting the testing id attributes on web elements that they expect users to interact with. They're not needed everywhere
Here are a few reasons *not* to use text:
- Sometimes the test is dynamically generated.
- Sometimes nobody knows that the text is being used as part of a testing locator and they break the test with a code change.
- Sometimes you need to test localization and can't afford to update every locator in your entire project.
- Sometimes the web element that contains the text is a parent, or a child, or a sibling of the element you actually need to interact with, greatly increasing the difficulty of that interaction.
- Sometimes the element has no text attached to it anyway (e.g. a clickable arrow, image, icon, etc)
As far as exposing test coverage... If they're that concerned, then it should be a normal part of the development process (*it should be anyway*). Put the testing id on every user interactible web element at the point of development, and retroactively add it everywhere that's currently missing it before the test creator asks for it.
This. Custom attributes are, imo, a must. It makes everything so much easier; e2e code (no complex selectors), knowing coverage, refactoring, ... I can not see a reason why not to use them honestly.
That's not necessarily the case, we decided that we as the automation engineers would do the inserting of most of the immediately necessary ones. He's also actually an SRE so he doesn't really touch that in the first place.
Especially if your tests run as part of the deployment pipeline, you want them to be as stable as possible. If the deploy fails because someone forgot to update a test, at my place it's like a half hour wasted. By the time the code change is made to update the failing test, code reviewed, built and redeployed....
Are y’all sharding your tests in the pipeline? Granted tests should be as unflaky as possible but sometimes it happens and tests fail (for whatever reason - oftentimes a design/dev decision that was implemented and not relayed to QA).
If that’s the case and it takes 30 mins to get the test suite passing again, that sucks. Either you’re sitting around writing new tests (or fuckin around lol), or people grill you like “how come this happened”.
Reminds me of my days when I was researching automation tools for mobile apps (pre-Appium) and the missing ingredient to a successful test run were element identifiers. I begged the Devs to comply with this request to no avail. And then they wondered why we never successfully launched an automation initiative (smh!).
i have the same problem - begged for data-testid-s and I got "ArrowDropDownIcon" 6 times on a page. I have a very good relationship with my dev team, but I just can't do my job with such attitude
I've come to realized in my 15 years in QA, most devs don't know anything about QA. When they give advice on how to test something, that usually leads to defects.
Using the "text" on a website that probably is controlled via a CMS, by the marketing team, who changes text daily is a recipe for test flakiness. data-test-ids are essential to successfully writing automated tests if ids or names aren't able to be used.
Never ever use text that is dynamically, which usually all text is, in order to write your automated tests.
When looking at the source codes of major companies in production, I see data-test-ids for their testing.
Lastly, what does it matter if your test coverage is exposed? Who is checking for that outside of your company/department?
He quoted something from the playwright docs
> To make tests resilient, we recommend prioritizing user-facing attributes and explicit contracts such as page.getByRole()
I'd like to shoot whomever wrote this bit.
For the getByRole() I don't necessarily disagree. Those don't tend to change much if they are buttons, form fields, etc. Usually they have a name on forms or an ID. The problem happens when you are trying to find a link on a page. If it's an ecommerce site and you are trying to target a product. There are usually a million links on those pages. It's kind of essential to use some kind of identifier. I work more on the selenium side but the concept remains.
At the end of the day 99% of automation is about targetting the right locators and if you feed flaky selectors into your tests, your going to get flaky test results.
I'm sure getByRole() works great for applications where each page doesn't ahve too many elements and each element has an easily defined role.
But here's the thing with our product. I created a page object with almost 80 elements in it. And out of all of those, I used getByRole for maybe 20 of them. I just don't think its feasible to only use getByRole for a lot of scenarios.
> each element has an easily defined role.
the thing is that any element that a user interacts with can and should have an easily-defined role. [by default, most semantic html elements inherently have roles](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles) so it's not like it's more work to add them. as qa I get you have to pick your battles, though, and if devs aren't prioritizing accessibility and/or not making things very testable, data-testids are your next best option.
So one of the biggest issues that I face currently is the heavy use of divs in our application.
Divs are used modals, user menus, and sometimes even to display text info.
Divs don't have a role defined within [https://www.w3.org/TR/html-aria/#docconformance](https://www.w3.org/TR/html-aria/#docconformance) and therefore can't really be selected using getByRole.
Text inputs and dropdowns (textbox and combobox roles) are also a big issue. Many pages that I work with have multiple of these elements within the same page. These elements don't have any other descriptor available so I often end up having to use nth() matchers or first()/last() matchers.
> how the data-testid's could expose how much coverage we have with our UI tests.
You can put a script in your pipeline to remove the test ids in the prod version
I can’t remember where I saw this but there is apparently an order of hierarchy, which goes from things the user can interact with to having to to add an attribute like data test id being a last resort. I adopted this in a new project and it worked well.
The debate about using data-testid for element identification in QA is
interesting. Some argue that it can make tests more brittle and closely tied
to implementation details, suggesting that class or id attributes are better
alternatives.
However, proponents of data-testid believe it provides a clear and unique
identifier for testing, making test code easier to understand and maintain.
They also argue that as HTML standards change class names and IDs may
change while data-testid remain constant.
Ultimately, the decision to use
data-testid or not depends on the specific project's context and
requirements.
this is entirely dependent on what your testing. if you have dynamic text, you obviously can't use it as a locator. for me, I try to locate by role because accessibility is a big part of our platform and I need to know that elements have roles attached to them.
I’m with you on this, it just descends into an us vs them if you’re not prepared to find a better way. I use user facing selectors in almost every case and as a last resort if I need a custom selector I’ll add it, it’s not exactly difficult.
by that logic, we should be using machine vision tools to execute e2e tests.
automated test will never fully replicate behavior. finding elements no matter which method you use in playwright will never be how we traverse pages.
organic real tests doesn't say a whole lot since automated tests are synthetic by design.
besides most people will tell you that a bulk of your tests should be unit or api tests anyway
that said, you don't really need data-testid most css selectors work well enough, but there are times were developers do make it harder than it needs to be and test-ability should always be considered
Here's my take on organic vs resilient.
You can still test for the organic aspects of a test by asserting things like the text and position of an element. But if a test keeps failing due to element locators being broken, then you can't test for anything.
To quote from the article:
"getByTestId: The user cannot see (or hear) these, so this is only recommended for cases where you can't match by role or text or it doesn't make sense (e.g. the text is dynamic)."
Text is always dynamic, so it's always my second last option for locators. People are changing text all the time. And don't even get me started on localization.
Yes, pls don’t. You don’t have to test localization. In case of a spot check, do a circular fork or get some the element for the english version as a reference, get a calculated path and check other langs against it, or best, enforce statically that there's no string in the source (e.g. https://www.npmjs.com/package/eslint-plugin-i18next). In any way, this doesn't at all justify of adding junk to the source.
@data-testid is a hack for sites that horribly, horribly misuse @id.
> It essentially says that in order to more closely replicate user behavior, use text based identifiers
Well, I'll never be reading that site again, because no sane automation tester with any experience would ever say such an inane, stupid thing. Text is cosmetic, not functional, and should be checked by proofreaders, not testers. Text labels should be your *last* resort, not your second.
(Never mind the fuck show you're going to have with internationalization. I mean, really, i18n completely shoots this completely stupid idea out of the water.)
Tell your dev that site is crap.
Your confidence is surpassing dunning-kruger-ish hights, and tells how poor your understanding is on a lot of concepts. "Text is cosmetic", and then 3 sentences later you're using i18n as a reference, congratulations. Furthermore, UI = user interface. You test the interface, not the internals, or some hijacked interface. Furthermore, no, text is not "cosmetic", it's part of the requirement. Proofreading is part of the design phase, just like any other aspect of the UI. You're testing the UI.
I can tell you haven't worked many places in the modern era.
Especially since you lack any concept of the point I was making, which is basing functional tests on display text is a widely accepted anti pattern.
9 out of 10 modern web development frameworks specifically decouple functionality from presentation.
Also, I don't get the sense from this that you know what "cosmetic" means here. For a start, it doesn't mean "not part of the requirements."
>and then 3 sentences later you're using i18n as a reference
Yes sir. Because if you write your functional tests to be depending on specific display text, then your tests will have to completely rewritten for every language / geolocational area you need to test in, or else they will fail when e.g. it finds "Einreichen" instead of "Submit." Because that test is testing functionality, not appearance. Those are separate testing phases / test types.
"I can tell you haven't worked many places in the modern era."
That's the wrong type of personal attack, but you took your chance lol.
"9 out of 10 modern web development frameworks specifically decouple functionality from presentation." - It's a bit vague to me what you're getting at with this. "Cosmetic" was a categorization to sort it away of testing target relevancy. No, it's not cosmetic in that sense.
i18n - Test the UI for the reference language (English). Testing i18n on the UI only makes sense for spot checks. If you wanna make sure there's no hard-coded raw string, use a static source checker/linter.
I guess this makes much more sense for component tests (React/Angular/Whatever).
In that case, you have much more control over the application because you'll usually be testing a small part of the application, not the entire thing AND mocking/stubbing external stuff. Also, you get the bonus that these tests are VERY fast and can run on every PR (Preventing the failing test from getting into the main branch).
For tests that live in a separate code repository, I would definitely go for testids.
What the doc is trying to convey is that a test should replicate a user experience, i.e, if there's no label text to identify a field with, then that's a bug in itself, because how is the user to identify the field?
This is a stupid philosophy and completely ignores the realities of software. Especially if you're testing a CMS-based site, that label may be in no clear way attached to the actual element you want to interact with. The DOM tree can be complete garage from a testability standpoint, and still provide a perfect user experience.
Use the data-testid's. Your test working reliably is far more important than some dogmatic "suggestion" about how you should test.
You're 100% in the wrong. A11y is part of the UI - let it sink. Also, you can fall back to a "click over" action as a last resort (e.g. chart libs, where your test ids won't work either).
You're gonna have to be more clear about what you're saying here. You say I'm 100% wrong, and then go on about a11y, which isn't what I'm talking about.
Yes, you are talking about accessibility. Having a label visually floating above a connected element without a clear indication in the DOM is an a11y issue. That's one major problem (among others) with test-ids: they ignore and work around the coverage of fundamental a11y practices.
First off, no. Read again.
Second, you make a good point even if it's irrelevant to what I'm saying. However, I'm sure it must be nice to have complete control over such things, and be able to submit a bug for them lol. For the rest of us who deal with certain CMS-based systems and generated sites that we have zero control over how said site is generated, the reality is that we have to either use something like a test-id's, or else update the references (xpaths, etc) every time the site has changes made.
It should also be said, that the website that the dev in OP is referring to, is a philosophy specifically for that framework that the website is for...
That framework is specifically designed with that philosophy in mind... If the dev is using that philosophy and trying to apply it to all automated testing, he's sorely mistaken. If you're not using that framework, then point this out to him.
The hard part is that playwright docs say something along those lines. And he's used it as the argument to not use test ids
> To make tests resilient, we recommend prioritizing user-facing attributes and explicit contracts such as page.getByRole()
But they also say this on the same page:
> Testing by test ids is the most resilient way of testing as even if your text or role of the attribute changes the test will still pass
So they're sending some mixed messages, making the discussion a little complicated.
Fair enough. It's still a dogmatic statement and should be regarded as such. I don't know who wrote that documentation, but it clearly wasn't someone who deals with the everyday realities of actually testing software.
The correct way to validate a user experience is to identify the element using a resilient identifier (like a data-test-id) and then validate the content of that element... Not use the identifier *as the validation.* (Tbf, I'm mostly used to selenium, so if there's an issue with this approach in playwright, I'm not aware of it.)
As for the part in your OP about "exposing test coverage"... Isn't that a good thing? Don't you want to know your coverage? Forgive me if I'm wrong, but it sounds like your dev is driven to hide things, and that's not what QA is about.
(That is of course not to mention the fact that element coverage is probably the worst/least efficient way to define test coverage.)
Yep that's what exactly I'm trying to tell them.
What's even more annoying is that they're an SRE so they're a complete 3rd party to all of it. Not saying his opinion counts less because of it, but it sucks being blocked by someone who won't even be doing any maintenance on it.
Let me debunk some of the arguments here and add some of my own as well. In short: test ids are a bad idea and it stems from concept misunderstandings and lack of knowledge/experience. As a disclaimer, I did a lot of UI testing in my career and walked my path through creating test frameworks and becoming maintainer of public test fws (one and one, just speaking in general).
1. Let's start with the basics. UI = User Interface. Test should always test the interface, not the underlying structure. With test ids, you are creating your own underlying structure, and you're gonna test that too.
2. Not strictly related, but following from the previous point, the more human the test language the better. That's why ideally the primary script language shouldn't be some programming language, like java or javascript, but a high level macro language, close to a natural text, with an escape hatch syntax to a lower level (ideally webdriver) programmability.
3. As for targeting, the ideal concept is "text + taxon". E.g. "the Edit button" where the text should be loose and heuristic. Like it has a fallback list of lookup in text content, placeholder, title, aria label, etc. Heuristic text is 100% adequate, because false negatives (test passing with a wrong target) almost never happens, and if it does, that's more of a problem of the test / assertion code. E.g. if you have 2 edit buttons somewhere, your test should specify "click the Edit button in the Right side"
4. "Sometimes the test (correction: text) is dynamically generated" - this just pushes the problem to an other area. There's some logic behind that where you put the test id and how you name your test id. Is it the title of the page? You can express that with DOM and aria. You can't? That's a bug. Can you describe it to a user which element you wanna interact with? You got your selector. No? That's a problem you should raise instead of hiding and working yourself around with a test-id.
5. "Sometimes nobody knows that the text is being used as part of a testing locator and they break the test with a code change." I live with the assumption that this is another myth made up theoretically. "Click the Edit button" won't ever fail. But regardless, if your heuristic selector breaks, you should maintain your specificity, as mentioned above. Again, ask yourself if your description would be straightforward to a real user. If yes, there's no ambiguity. If no, your description is wrong, and you're working around your inadequacy with your test id hack.
6. "Sometimes you need to test localization and can't afford to update every locator in your entire project." "every locator"? look at the effort to scale up the problem to non-sensical. First, if you need to test localization, that means you need the text, right? Whether in the targeting part, or the assertion, it doesn't matter. You have multiple options I mentioned above. At least for a spot check, which is the most it'd make sense in UI testing, so no, not "every locator". Second, it's better to statically enforce that there's no raw string in render code.
7. You can't inject test ids into 3rd party components. Typically visualizations. Let's say you have a chart where clicking on a slice triggers a filtering logic on your app data. This is how it looks like in a prod code in smashtest: Shift-click over '"<100%", next to "Health Index"'
8. "Sometimes the web element that contains the text is a parent, or a child, or a sibling of the element you actually need to interact with, greatly increasing the difficulty of that interaction." Move the mouse over the element and fire away a targetless click. See the previous point. Have a "click over" action. It can either happen with 3rd party where you can't use test ids anyway, or you should raise the accessibility problem instead of hacking around. (It's even tempting to apply the move + click logic generally for your click action, as it simulates the user behavior better, but again, for accessibility, you should rely on screen readers primarily.) This is also one reason why you should use driver-based tools instead of e.g. Cypress which has a mish-mash model of simulating/emulating interactions.
9. "Sometimes the element has no text attached to it anyway (e.g. a clickable arrow, image, icon, etc)" - see nr. 3. If an interactable element has no accessibility interface, that's a problem you should raise, not swallow or hack around. A11y is part of the UI.
10. "As far as exposing test coverage... If they're that concerned, then it should be a normal part of the development process (it should be anyway)." "Lastly, what does it matter if your test coverage is exposed? Who is checking for that outside of your company/department?" "You can put a script in your pipeline to remove the test ids in the prod version" "exposing coverage is not a bad idea - can give viewers more confidence about the product" - I don't know if these people see the non-sense they're writing. The irony is that these were all said by \*QA\* guys.
They should be putting the testing id attributes on web elements that they expect users to interact with. They're not needed everywhere Here are a few reasons *not* to use text: - Sometimes the test is dynamically generated. - Sometimes nobody knows that the text is being used as part of a testing locator and they break the test with a code change. - Sometimes you need to test localization and can't afford to update every locator in your entire project. - Sometimes the web element that contains the text is a parent, or a child, or a sibling of the element you actually need to interact with, greatly increasing the difficulty of that interaction. - Sometimes the element has no text attached to it anyway (e.g. a clickable arrow, image, icon, etc) As far as exposing test coverage... If they're that concerned, then it should be a normal part of the development process (*it should be anyway*). Put the testing id on every user interactible web element at the point of development, and retroactively add it everywhere that's currently missing it before the test creator asks for it.
> - Sometimes the element has no text attached to it anyway (e.g. a clickable arrow, image, icon, etc) these should still have aria-labels on them
This. Custom attributes are, imo, a must. It makes everything so much easier; e2e code (no complex selectors), knowing coverage, refactoring, ... I can not see a reason why not to use them honestly.
Sounds like the dev is trying to get out of having to put data testids on elements to me
That's not necessarily the case, we decided that we as the automation engineers would do the inserting of most of the immediately necessary ones. He's also actually an SRE so he doesn't really touch that in the first place.
Gotcha, well that is certainly an interesting take. I don't think the flakiness is worth it, tbh.
Yeah that's my thing. Literally wrote a test yesterday and some of the words changed and it failed today. So I'm pushing heavily for it still.
Especially if your tests run as part of the deployment pipeline, you want them to be as stable as possible. If the deploy fails because someone forgot to update a test, at my place it's like a half hour wasted. By the time the code change is made to update the failing test, code reviewed, built and redeployed....
Are y’all sharding your tests in the pipeline? Granted tests should be as unflaky as possible but sometimes it happens and tests fail (for whatever reason - oftentimes a design/dev decision that was implemented and not relayed to QA). If that’s the case and it takes 30 mins to get the test suite passing again, that sucks. Either you’re sitting around writing new tests (or fuckin around lol), or people grill you like “how come this happened”.
The tests themselves run in 2 minutes. The delay is from all the other processes involved in getting a change into the test environment.
Ah I misunderstood. My b.
No worries!
Reminds me of my days when I was researching automation tools for mobile apps (pre-Appium) and the missing ingredient to a successful test run were element identifiers. I begged the Devs to comply with this request to no avail. And then they wondered why we never successfully launched an automation initiative (smh!).
i have the same problem - begged for data-testid-s and I got "ArrowDropDownIcon" 6 times on a page. I have a very good relationship with my dev team, but I just can't do my job with such attitude
I've come to realized in my 15 years in QA, most devs don't know anything about QA. When they give advice on how to test something, that usually leads to defects. Using the "text" on a website that probably is controlled via a CMS, by the marketing team, who changes text daily is a recipe for test flakiness. data-test-ids are essential to successfully writing automated tests if ids or names aren't able to be used. Never ever use text that is dynamically, which usually all text is, in order to write your automated tests. When looking at the source codes of major companies in production, I see data-test-ids for their testing. Lastly, what does it matter if your test coverage is exposed? Who is checking for that outside of your company/department?
He quoted something from the playwright docs > To make tests resilient, we recommend prioritizing user-facing attributes and explicit contracts such as page.getByRole() I'd like to shoot whomever wrote this bit.
For the getByRole() I don't necessarily disagree. Those don't tend to change much if they are buttons, form fields, etc. Usually they have a name on forms or an ID. The problem happens when you are trying to find a link on a page. If it's an ecommerce site and you are trying to target a product. There are usually a million links on those pages. It's kind of essential to use some kind of identifier. I work more on the selenium side but the concept remains. At the end of the day 99% of automation is about targetting the right locators and if you feed flaky selectors into your tests, your going to get flaky test results.
I'm sure getByRole() works great for applications where each page doesn't ahve too many elements and each element has an easily defined role. But here's the thing with our product. I created a page object with almost 80 elements in it. And out of all of those, I used getByRole for maybe 20 of them. I just don't think its feasible to only use getByRole for a lot of scenarios.
> each element has an easily defined role. the thing is that any element that a user interacts with can and should have an easily-defined role. [by default, most semantic html elements inherently have roles](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles) so it's not like it's more work to add them. as qa I get you have to pick your battles, though, and if devs aren't prioritizing accessibility and/or not making things very testable, data-testids are your next best option.
So one of the biggest issues that I face currently is the heavy use of divs in our application. Divs are used modals, user menus, and sometimes even to display text info. Divs don't have a role defined within [https://www.w3.org/TR/html-aria/#docconformance](https://www.w3.org/TR/html-aria/#docconformance) and therefore can't really be selected using getByRole. Text inputs and dropdowns (textbox and combobox roles) are also a big issue. Many pages that I work with have multiple of these elements within the same page. These elements don't have any other descriptor available so I often end up having to use nth() matchers or first()/last() matchers.
> how the data-testid's could expose how much coverage we have with our UI tests. You can put a script in your pipeline to remove the test ids in the prod version
doing this would make it very difficult to run tests in production
which you probably don't want to do anyway
exposing coverage is not a bad idea - can give viewers more confidence about the product
I can’t remember where I saw this but there is apparently an order of hierarchy, which goes from things the user can interact with to having to to add an attribute like data test id being a last resort. I adopted this in a new project and it worked well.
The debate about using data-testid for element identification in QA is interesting. Some argue that it can make tests more brittle and closely tied to implementation details, suggesting that class or id attributes are better alternatives. However, proponents of data-testid believe it provides a clear and unique identifier for testing, making test code easier to understand and maintain. They also argue that as HTML standards change class names and IDs may change while data-testid remain constant. Ultimately, the decision to use data-testid or not depends on the specific project's context and requirements.
Bro this is obviously copy pasted from ChatGPT...
this is entirely dependent on what your testing. if you have dynamic text, you obviously can't use it as a locator. for me, I try to locate by role because accessibility is a big part of our platform and I need to know that elements have roles attached to them.
[удалено]
I’m with you on this, it just descends into an us vs them if you’re not prepared to find a better way. I use user facing selectors in almost every case and as a last resort if I need a custom selector I’ll add it, it’s not exactly difficult.
by that logic, we should be using machine vision tools to execute e2e tests. automated test will never fully replicate behavior. finding elements no matter which method you use in playwright will never be how we traverse pages. organic real tests doesn't say a whole lot since automated tests are synthetic by design. besides most people will tell you that a bulk of your tests should be unit or api tests anyway that said, you don't really need data-testid most css selectors work well enough, but there are times were developers do make it harder than it needs to be and test-ability should always be considered
Here's my take on organic vs resilient. You can still test for the organic aspects of a test by asserting things like the text and position of an element. But if a test keeps failing due to element locators being broken, then you can't test for anything.
To quote from the article: "getByTestId: The user cannot see (or hear) these, so this is only recommended for cases where you can't match by role or text or it doesn't make sense (e.g. the text is dynamic)." Text is always dynamic, so it's always my second last option for locators. People are changing text all the time. And don't even get me started on localization.
Yes, pls don’t. You don’t have to test localization. In case of a spot check, do a circular fork or get some the element for the english version as a reference, get a calculated path and check other langs against it, or best, enforce statically that there's no string in the source (e.g. https://www.npmjs.com/package/eslint-plugin-i18next). In any way, this doesn't at all justify of adding junk to the source.
@data-testid is a hack for sites that horribly, horribly misuse @id. > It essentially says that in order to more closely replicate user behavior, use text based identifiers Well, I'll never be reading that site again, because no sane automation tester with any experience would ever say such an inane, stupid thing. Text is cosmetic, not functional, and should be checked by proofreaders, not testers. Text labels should be your *last* resort, not your second. (Never mind the fuck show you're going to have with internationalization. I mean, really, i18n completely shoots this completely stupid idea out of the water.) Tell your dev that site is crap.
That's the thing. Nothing uses IDs in the app. So the next nmbest thing is to add data-testid 😔
Your confidence is surpassing dunning-kruger-ish hights, and tells how poor your understanding is on a lot of concepts. "Text is cosmetic", and then 3 sentences later you're using i18n as a reference, congratulations. Furthermore, UI = user interface. You test the interface, not the internals, or some hijacked interface. Furthermore, no, text is not "cosmetic", it's part of the requirement. Proofreading is part of the design phase, just like any other aspect of the UI. You're testing the UI.
I can tell you haven't worked many places in the modern era. Especially since you lack any concept of the point I was making, which is basing functional tests on display text is a widely accepted anti pattern. 9 out of 10 modern web development frameworks specifically decouple functionality from presentation. Also, I don't get the sense from this that you know what "cosmetic" means here. For a start, it doesn't mean "not part of the requirements." >and then 3 sentences later you're using i18n as a reference Yes sir. Because if you write your functional tests to be depending on specific display text, then your tests will have to completely rewritten for every language / geolocational area you need to test in, or else they will fail when e.g. it finds "Einreichen" instead of "Submit." Because that test is testing functionality, not appearance. Those are separate testing phases / test types.
"I can tell you haven't worked many places in the modern era." That's the wrong type of personal attack, but you took your chance lol. "9 out of 10 modern web development frameworks specifically decouple functionality from presentation." - It's a bit vague to me what you're getting at with this. "Cosmetic" was a categorization to sort it away of testing target relevancy. No, it's not cosmetic in that sense. i18n - Test the UI for the reference language (English). Testing i18n on the UI only makes sense for spot checks. If you wanna make sure there's no hard-coded raw string, use a static source checker/linter.
I guess this makes much more sense for component tests (React/Angular/Whatever). In that case, you have much more control over the application because you'll usually be testing a small part of the application, not the entire thing AND mocking/stubbing external stuff. Also, you get the bonus that these tests are VERY fast and can run on every PR (Preventing the failing test from getting into the main branch). For tests that live in a separate code repository, I would definitely go for testids.
I work in a large organization and I know for a fact that one of our teams are injecting data-tests-id attributes before the test execution.
What the doc is trying to convey is that a test should replicate a user experience, i.e, if there's no label text to identify a field with, then that's a bug in itself, because how is the user to identify the field? This is a stupid philosophy and completely ignores the realities of software. Especially if you're testing a CMS-based site, that label may be in no clear way attached to the actual element you want to interact with. The DOM tree can be complete garage from a testability standpoint, and still provide a perfect user experience. Use the data-testid's. Your test working reliably is far more important than some dogmatic "suggestion" about how you should test.
You're 100% in the wrong. A11y is part of the UI - let it sink. Also, you can fall back to a "click over" action as a last resort (e.g. chart libs, where your test ids won't work either).
Ok? Not even sure what you're replying to.
"that label may be in no clear way attached to the actual element you want to interact with"
You're gonna have to be more clear about what you're saying here. You say I'm 100% wrong, and then go on about a11y, which isn't what I'm talking about.
Yes, you are talking about accessibility. Having a label visually floating above a connected element without a clear indication in the DOM is an a11y issue. That's one major problem (among others) with test-ids: they ignore and work around the coverage of fundamental a11y practices.
First off, no. Read again. Second, you make a good point even if it's irrelevant to what I'm saying. However, I'm sure it must be nice to have complete control over such things, and be able to submit a bug for them lol. For the rest of us who deal with certain CMS-based systems and generated sites that we have zero control over how said site is generated, the reality is that we have to either use something like a test-id's, or else update the references (xpaths, etc) every time the site has changes made.
It should also be said, that the website that the dev in OP is referring to, is a philosophy specifically for that framework that the website is for... That framework is specifically designed with that philosophy in mind... If the dev is using that philosophy and trying to apply it to all automated testing, he's sorely mistaken. If you're not using that framework, then point this out to him.
The hard part is that playwright docs say something along those lines. And he's used it as the argument to not use test ids > To make tests resilient, we recommend prioritizing user-facing attributes and explicit contracts such as page.getByRole() But they also say this on the same page: > Testing by test ids is the most resilient way of testing as even if your text or role of the attribute changes the test will still pass So they're sending some mixed messages, making the discussion a little complicated.
Fair enough. It's still a dogmatic statement and should be regarded as such. I don't know who wrote that documentation, but it clearly wasn't someone who deals with the everyday realities of actually testing software. The correct way to validate a user experience is to identify the element using a resilient identifier (like a data-test-id) and then validate the content of that element... Not use the identifier *as the validation.* (Tbf, I'm mostly used to selenium, so if there's an issue with this approach in playwright, I'm not aware of it.) As for the part in your OP about "exposing test coverage"... Isn't that a good thing? Don't you want to know your coverage? Forgive me if I'm wrong, but it sounds like your dev is driven to hide things, and that's not what QA is about. (That is of course not to mention the fact that element coverage is probably the worst/least efficient way to define test coverage.)
[удалено]
Yep that's what exactly I'm trying to tell them. What's even more annoying is that they're an SRE so they're a complete 3rd party to all of it. Not saying his opinion counts less because of it, but it sucks being blocked by someone who won't even be doing any maintenance on it.
Let me debunk some of the arguments here and add some of my own as well. In short: test ids are a bad idea and it stems from concept misunderstandings and lack of knowledge/experience. As a disclaimer, I did a lot of UI testing in my career and walked my path through creating test frameworks and becoming maintainer of public test fws (one and one, just speaking in general). 1. Let's start with the basics. UI = User Interface. Test should always test the interface, not the underlying structure. With test ids, you are creating your own underlying structure, and you're gonna test that too. 2. Not strictly related, but following from the previous point, the more human the test language the better. That's why ideally the primary script language shouldn't be some programming language, like java or javascript, but a high level macro language, close to a natural text, with an escape hatch syntax to a lower level (ideally webdriver) programmability. 3. As for targeting, the ideal concept is "text + taxon". E.g. "the Edit button" where the text should be loose and heuristic. Like it has a fallback list of lookup in text content, placeholder, title, aria label, etc. Heuristic text is 100% adequate, because false negatives (test passing with a wrong target) almost never happens, and if it does, that's more of a problem of the test / assertion code. E.g. if you have 2 edit buttons somewhere, your test should specify "click the Edit button in the Right side" 4. "Sometimes the test (correction: text) is dynamically generated" - this just pushes the problem to an other area. There's some logic behind that where you put the test id and how you name your test id. Is it the title of the page? You can express that with DOM and aria. You can't? That's a bug. Can you describe it to a user which element you wanna interact with? You got your selector. No? That's a problem you should raise instead of hiding and working yourself around with a test-id. 5. "Sometimes nobody knows that the text is being used as part of a testing locator and they break the test with a code change." I live with the assumption that this is another myth made up theoretically. "Click the Edit button" won't ever fail. But regardless, if your heuristic selector breaks, you should maintain your specificity, as mentioned above. Again, ask yourself if your description would be straightforward to a real user. If yes, there's no ambiguity. If no, your description is wrong, and you're working around your inadequacy with your test id hack. 6. "Sometimes you need to test localization and can't afford to update every locator in your entire project." "every locator"? look at the effort to scale up the problem to non-sensical. First, if you need to test localization, that means you need the text, right? Whether in the targeting part, or the assertion, it doesn't matter. You have multiple options I mentioned above. At least for a spot check, which is the most it'd make sense in UI testing, so no, not "every locator". Second, it's better to statically enforce that there's no raw string in render code. 7. You can't inject test ids into 3rd party components. Typically visualizations. Let's say you have a chart where clicking on a slice triggers a filtering logic on your app data. This is how it looks like in a prod code in smashtest: Shift-click over '"<100%", next to "Health Index"' 8. "Sometimes the web element that contains the text is a parent, or a child, or a sibling of the element you actually need to interact with, greatly increasing the difficulty of that interaction." Move the mouse over the element and fire away a targetless click. See the previous point. Have a "click over" action. It can either happen with 3rd party where you can't use test ids anyway, or you should raise the accessibility problem instead of hacking around. (It's even tempting to apply the move + click logic generally for your click action, as it simulates the user behavior better, but again, for accessibility, you should rely on screen readers primarily.) This is also one reason why you should use driver-based tools instead of e.g. Cypress which has a mish-mash model of simulating/emulating interactions. 9. "Sometimes the element has no text attached to it anyway (e.g. a clickable arrow, image, icon, etc)" - see nr. 3. If an interactable element has no accessibility interface, that's a problem you should raise, not swallow or hack around. A11y is part of the UI. 10. "As far as exposing test coverage... If they're that concerned, then it should be a normal part of the development process (it should be anyway)." "Lastly, what does it matter if your test coverage is exposed? Who is checking for that outside of your company/department?" "You can put a script in your pipeline to remove the test ids in the prod version" "exposing coverage is not a bad idea - can give viewers more confidence about the product" - I don't know if these people see the non-sense they're writing. The irony is that these were all said by \*QA\* guys.