[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [sarif] RE: your thoughts on DARIF
Although dynamic assessments can and do produce data that is not expressible in the current version of SARIF, there seems to be a large intersection between the data necessary to express static analysis and dynamic analysis results, and unnecessary data can simply be omitted by producers. Having a single format makes life simpler for consumers and SARIF library writers. Ideally I would like to see a unified format that expands SARIF to add the features necessary to fully support dynamic assessment tools. I think we should first look at enhancing SARIF, but allow for the possibility of producing a separate DARIF. To make this determination, the first step of the process would be to enumerate the missing pieces of data, and determine if SARIF would become too unwieldy if there were added as run object properties or elsewhere. Adding the following should go a long way to expressing the needs of running and the results of many dynamic analysis tools: 1) generic metric data to SARIF (necessary for static analysis tools also), 2) a way to describe the set of tests performed (the run-time environment and test input data provided), and 3) correlating binary artifacts to source and how it was built should go a long way to expressing the needs of running and the results of many dynamic analysis tools. For instance valgrind results fit easy into SARIF 2.1 (there is a run, a tool, results with locations, ...) except there is no way to express 2 and 3 in SARIF. Jim On 7/23/19 11:48 PM, Yekaterina O'Neil wrote: > Thank you all for so enthusiastically responding! > > I, however, would still argue that DARIF should be separate from SARIF. > In Fortify’s experience, what you need to describe dynamic results is > very different from what you need to describe static results. And I > would argue against extending SARIF, especially considering there is a > talk of incorporating other types of analyses. I am wondering, however, > whether it would be possible to create a suite of standards under the > same umbrella and structure it in such a way that one is able to > implement/use part of the suite according to their needs. Is it possible > to create something like a Program Analysis Results Interchange Format > (PARIF) that contains sections for SARIF, DARIF, CCARIF (“code > coverage”), MARIF (“metrics”) , etc.? > > Thoughts? > > k > > *From:*sarif@lists.oasis-open.org [mailto:sarif@lists.oasis-open.org] > *On Behalf Of *Larry Golding (Myriad Consulting Inc) > *Sent:* Friday, July 19, 2019 4:42 PM > *To:* Michael Fanning <Michael.Fanning@microsoft.com>; Paul Anderson > <paul@grammatech.com>; Yekaterina O'Neil <katrina@microfocus.com>; > sarif@lists.oasis-open.org > *Cc:* David Keaton <dmk@dmk.com> > *Subject:* RE: [sarif] RE: your thoughts on DARIF > > Actually, if we can maintain back compat (only adding to the schema, no > changes or deletions) then according to Semver <https://semver.org/> > DARIF could go out as a /minor/ revision 2.2.0: > > ØBug fixes not affecting the API increment the patch version, backwards > compatible API additions/changes increment the minor version, and > backwards incompatible API changes increment the major version. > > That would go down much more easily for existing customers of SARIF 2.1.0. > > And it wouldn’t prevent us from bug-fixing the current spec as 2.1.1. > > Larry > > *From:*Michael Fanning <Michael.Fanning@microsoft.com > <mailto:Michael.Fanning@microsoft.com>> > *Sent:* Friday, July 19, 2019 4:35 PM > *To:* Paul Anderson <paul@grammatech.com <mailto:paul@grammatech.com>>; > Larry Golding (Myriad Consulting Inc) <v-lgold@microsoft.com > <mailto:v-lgold@microsoft.com>>; Yekaterina O'Neil > <katrina@microfocus.com <mailto:katrina@microfocus.com>>; > sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> > *Cc:* David Keaton <dmk@dmk.com <mailto:dmk@dmk.com>> > *Subject:* RE: [sarif] RE: your thoughts on DARIF > > It seems like we’ve got the nucleus of an interesting moving forward plan. > > 1)Invoke the OASIS machinery to expand the charter of this technical > committee to include dynamic analysis results/metrics. > > 2)As part of this process, remap the ‘S’ in SARIF to ‘Software’ to > retain the brand. Bless the new proposed standard as a v3 effort to > indicate it is a major revision. > > 3)Explore moving forward funding/staffing of the technical committee. > > David pointed out offline that convening a moving forward v3 effort > might also create more flexibility in bug-fixing the v2 spec along the > way (a point I agree with). > > I will float this idea around MS a bit. > > On a separate note, a different team approached me on producing a > standard that attempts to codify a complete data model for end-to-end > engineering, including static analysis. Think models to express > projects, people, release pipelines, deployment environments, analysis > results, etc. etc. and linking them together. For quality, compliance, > auditing, chain-of-custody purposes. > > Michael > > *From:*Paul Anderson <paul@grammatech.com <mailto:paul@grammatech.com>> > *Sent:* Friday, July 19, 2019 9:24 AM > *To:* Larry Golding (Myriad Consulting Inc) <v-lgold@microsoft.com > <mailto:v-lgold@microsoft.com>>; Michael Fanning > <Michael.Fanning@microsoft.com <mailto:Michael.Fanning@microsoft.com>>; > Yekaterina O'Neil <katrina@microfocus.com > <mailto:katrina@microfocus.com>>; sarif@lists.oasis-open.org > <mailto:sarif@lists.oasis-open.org> > *Subject:* Re: [sarif] RE: your thoughts on DARIF > > All: > > We're very interested too. We've had some success with expressing > dynamic properties in SARIF already. I would argue that we should work > to have one standard that can satisfy both needs. Perhaps the 'S' should > stand for just 'Software'. Having said that, someone here is also > looking at using it to express properties in HDLs. > > To express test coverage results it will be important to express metrics > in the format. For example, a commonly used metric is the Test > Effectiveness Ratio (TER), which is percentage of the function that got > covered by the test suite. It would be super useful to be able to > express that in SARIF. (Some organizations have rules that require TER > to be greater than some threshold, and violations of that rule can then > be expressed as SARIF results.) Many other dynamic results (e.g., > profiling) are also expressed as metrics, so I believe that's the > biggest gap in SARIF for dynamic use. > > -Paul > > On 7/18/2019 6:53 PM, Larry Golding (Myriad Consulting Inc) wrote: > > Yes, I would also be very interested in exploring this. > > Larry > > *From:* sarif@lists.oasis-open.org > <mailto:sarif@lists.oasis-open.org> <sarif@lists.oasis-open.org> > <mailto:sarif@lists.oasis-open.org> *On Behalf Of *Michael Fanning > *Sent:* Thursday, July 18, 2019 3:51 PM > *To:* Yekaterina O'Neil <katrina@microfocus.com> > <mailto:katrina@microfocus.com>; sarif@lists.oasis-open.org > <mailto:sarif@lists.oasis-open.org> > *Subject:* [sarif] RE: your thoughts on DARIF > > Microsoft would definitely be interested in exploring this > possibility. I was just pinged this week on the possibilities using > SARIF for capturing code coverage results (which are apparently of > significant compliance interest for the automotive industry). This > would be a good ‘seed’ scenario for a dynamic analysis initiative > (along with profiling, web testing, etc.). > > Pull us in when you’re ready to talk. I will help connect the > conversation with MS teams who are stakeholders in these areas. > > https://en.wikipedia.org/wiki/ISO_26262 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FISO_26262&data=02%7C01%7Cv-lgold%40microsoft.com%7Cfa961fa69a604b9f9ed008d70ca1ad4e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636991760851525139&sdata=gfCMFDwJsFCodIaepYRapT%2BSGUS1RmSBlwSUFjSFRyY%3D&reserved=0> > > *From:* sarif@lists.oasis-open.org > <mailto:sarif@lists.oasis-open.org> <sarif@lists.oasis-open.org > <mailto:sarif@lists.oasis-open.org>> *On Behalf Of *Yekaterina O'Neil > *Sent:* Thursday, July 18, 2019 11:16 AM > *To:* sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> > *Subject:* [sarif] your thoughts on DARIF > > Hi all, > > As we are hopefully approaching the release of the first version of > the SARIF standard, I wanted to bring up the conversation we had > earlier about standardizing dynamic results. I was curious who on > this list would potentially be interested in pursuing a DARIF some > time in the future. > > Thanks! > > k > > -- > > Paul Anderson, VP of Engineering, GrammaTech, Inc. > > 531 Esty St., Ithaca, NY 14850 > > Tel: +1 607 273-7340 x118;http://www.grammatech.com <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.grammatech.com&data=02%7C01%7Cv-lgold%40microsoft.com%7Cfa961fa69a604b9f9ed008d70ca1ad4e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636991760851535137&sdata=IlFiKIhw6kBxw3MbUEVUDFGNQVaBQaHhT44tIplz4U0%3D&reserved=0> >
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]