Choosing an AppSec platform: a checklist of questions for the vendor
A demo almost always looks good. The difference shows on your code, your process and your perimeter. Below are questions worth asking any vendor, including us, and how to run a pilot so the comparison is fair.
TRAIO Insights material · regulator requirements are covered in separate reviews ARDFM No. 53 and Unified Requirements, with quotes of the clauses
Checklist
- Coverage checked on your stack, not from the vendor's list
- Pilot on your own repository: the share of real findings is counted
- Duplicates and false positives are removed, triage marks are kept
- Prioritization takes exploitability into account, not only CVSS
- Connects without rewriting CI/CD, the result appears in the merge request
- Quality gate with thresholds and an observation mode
- On-Premise in an isolated environment, code does not leave the perimeter
- Check history by version and reports for the auditor
- Total cost is clear: license + people + support
- Data can be exported if you switch tools
1. What exactly the platform checks
Start from your own stack, not from the vendor's presentation. Write down languages, frameworks, mobile apps, containers, infrastructure as code, and check coverage against that list.
- Which kinds of checks are there: SAST, SCA, DAST, secret detection, IaC, containers, mobile apps?
- Which languages does SAST support, on your code and not by the list on the website? Is there 1C, if you have it?
- How many analyzers are inside, and can you plug in your own if you already have a favorite tool?
- How does the platform build an SBOM and in which formats (CycloneDX, SPDX)?
2. How much noise and how many false positives
The main reason AppSec tools stop being used is noise. A thousand findings with no priority is not a result but a new problem.
- Does the platform remove duplicates between different analyzers and between scans?
- How does it prioritize: only by CVSS, or does it take exploitability (KEV, EPSS) and application context into account?
- Can a false positive be marked once so that it does not return?
- Ask for a pilot on your own repository and count the share of real findings among the first 50.
3. How it fits into development
A good tool fits into the developer's usual process. A bad one forces you to rewrite pipelines and go to a separate console.
- Do you need to change CI/CD, or does the platform connect to the repository by itself?
- Where does the developer see the result: a comment in the merge request, a task in the tracker, the IDE?
- Can a quality gate with severity thresholds be configured, first in observation mode, then blocking?
- Does it work with your version control system (GitLab, GitHub, Bitbucket) and tracker?
4. Where the code and data are stored
For banks and the public sector this is often the deciding question: source code and check results must not leave the perimeter.
- Is there an On-Premise installation in an isolated environment with no internet access?
- How are rules and vulnerability databases updated in an isolated environment?
- What are the hardware requirements and how long does installation take?
- Are code, metadata or telemetry sent anywhere?
5. Regulator requirements and reporting
If you are a bank or work with the public sector, the platform should help prove compliance, not only find vulnerabilities.
- Does it cover the mandatory static analysis and library analysis for remote service software (clauses 148–150 of ARDFM No. 53)?
- Does it keep check history by version? The resolution requires access to testing results for 3 years (clause 153).
- Are there reports for management and the auditor: by application, severity, time to fix?
- For the public sector: will it help prepare for testing for information security compliance (clause 87 of the Unified Requirements)?
6. Who will use it
Count not only the license but also the people: who configures, who triages findings, who maintains.
- How much time per week does an AppSec engineer spend triaging results?
- Is there support in Russian, and what is the response time?
- How is the cost calculated: per developer, per application, per scan, and what happens as you grow?
- Can all data be exported if you decide to switch tools?
Questions and answers
Where to start choosing an AppSec platform?
With an inventory of your stack and requirements: languages, application types, where the code must be stored, which regulator norms apply. Then a short list and pilots on your own code.
How long should a pilot last?
Usually 2–4 weeks on 2–3 real repositories. That is enough to assess coverage, noise and developer convenience.
Is SAST alone enough?
No. SAST checks your code but does not see vulnerable libraries (SCA), configuration errors (IaC, containers) or problems in a running application (DAST). For banks in Kazakhstan, library analysis is mandatory alongside static analysis.
What matters more, the number of analyzers or the quality of results?
The quality of results. Many analyzers help only if the platform merges their findings into one list without duplicates and with a clear priority.