[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [sarif] RE: your thoughts on DARIF
I'm very interested to hear what dynamic analysis properties could not be made to fit SARIF other than the few we've already mentioned.
-Paul On 7/25/2019 1:00 AM, Yekaterina O'Neil wrote:
Well, as I said earlier, I vote for DARIF being a separate standard, and thus refactoring SARIF/DARIF commonality into a BARIF k -----Original Message----- From: sarif@lists.oasis-open.org [mailto:sarif@lists.oasis-open.org] On Behalf Of Larry Golding (Myriad Consulting Inc) Sent: Wednesday, July 24, 2019 3:41 PM To: Yekaterina O'Neil <katrina@microfocus.com>; James Kupsch <kupsch@cs.wisc.edu>; sarif@lists.oasis-open.org Subject: RE: [sarif] RE: your thoughts on DARIF There are two questions: - Is DARIF a separate standard? Jim suggests we explore that idea ("I think we should first look at enhancing SARIF, but allow for the possibility of producing a separate DARIF.") It might or might not end up being separate. - _If_ DARIF is a separate standard, do we refactor the DARIF/SARIF commonality into a BARIF? I do think that would be a good thing -- but again, only if DARIF is a separate standard from SARIF, which is TBD. -----Original Message----- From: sarif@lists.oasis-open.org <sarif@lists.oasis-open.org> On Behalf Of Yekaterina O'Neil Sent: Wednesday, July 24, 2019 2:09 PM To: James Kupsch <kupsch@cs.wisc.edu>; sarif@lists.oasis-open.org Subject: RE: [sarif] RE: your thoughts on DARIF I think that what Larry proposed by collecting all the shared information in BARIF and defining the family of other standards that extend it in various ways makes a lot of sense and should suffice both of our needs. k -----Original Message----- From: sarif@lists.oasis-open.org [mailto:sarif@lists.oasis-open.org] On Behalf Of James Kupsch Sent: Wednesday, July 24, 2019 12:05 PM To: sarif@lists.oasis-open.org Subject: Re: [sarif] RE: your thoughts on DARIF Although dynamic assessments can and do produce data that is not expressible in the current version of SARIF, there seems to be a large intersection between the data necessary to express static analysis and dynamic analysis results, and unnecessary data can simply be omitted by producers. Having a single format makes life simpler for consumers and SARIF library writers. Ideally I would like to see a unified format that expands SARIF to add the features necessary to fully support dynamic assessment tools. I think we should first look at enhancing SARIF, but allow for the possibility of producing a separate DARIF. To make this determination, the first step of the process would be to enumerate the missing pieces of data, and determine if SARIF would become too unwieldy if there were added as run object properties or elsewhere. Adding the following should go a long way to expressing the needs of running and the results of many dynamic analysis tools: 1) generic metric data to SARIF (necessary for static analysis tools also), 2) a way to describe the set of tests performed (the run-time environment and test input data provided), and 3) correlating binary artifacts to source and how it was built should go a long way to expressing the needs of running and the results of many dynamic analysis tools. For instance valgrind results fit easy into SARIF 2.1 (there is a run, a tool, results with locations, ...) except there is no way to express 2 and 3 in SARIF. Jim On 7/23/19 11:48 PM, Yekaterina O'Neil wrote:Thank you all for so enthusiastically responding! I, however, would still argue that DARIF should be separate from SARIF. In Fortify's experience, what you need to describe dynamic results is very different from what you need to describe static results. And I would argue against extending SARIF, especially considering there is a talk of incorporating other types of analyses. I am wondering, however, whether it would be possible to create a suite of standards under the same umbrella and structure it in such a way that one is able to implement/use part of the suite according to their needs. Is it possible to create something like a Program Analysis Results Interchange Format (PARIF) that contains sections for SARIF, DARIF, CCARIF ("code coverage"), MARIF ("metrics") , etc.? Thoughts? k *From:*sarif@lists.oasis-open.org [mailto:sarif@lists.oasis-open.org] *On Behalf Of *Larry Golding (Myriad Consulting Inc) *Sent:* Friday, July 19, 2019 4:42 PM *To:* Michael Fanning <Michael.Fanning@microsoft.com>; Paul Anderson <paul@grammatech.com>; Yekaterina O'Neil <katrina@microfocus.com>; sarif@lists.oasis-open.org *Cc:* David Keaton <dmk@dmk.com> *Subject:* RE: [sarif] RE: your thoughts on DARIF Actually, if we can maintain back compat (only adding to the schema, no changes or deletions) then according to Semver <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsemver.org%2F&data=02%7C01%7Cv-lgold%40microsoft.com%7C17889714dc9d410bae0a08d7107b5f8b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636995994388361434&sdata=YKDI6VcoReMr50aJ%2FSey4%2FO%2FCTPLlX2E6r2ZUYk0I4Y%3D&reserved=0> DARIF could go out as a /minor/ revision 2.2.0: ÃBug fixes not affecting the API increment the patch version, backwards compatible API additions/changes increment the minor version, and backwards incompatible API changes increment the major version. That would go down much more easily for existing customers of SARIF 2.1.0. And it wouldn't prevent us from bug-fixing the current spec as 2.1.1. Larry *From:*Michael Fanning <Michael.Fanning@microsoft.com <mailto:Michael.Fanning@microsoft.com>> *Sent:* Friday, July 19, 2019 4:35 PM *To:* Paul Anderson <paul@grammatech.com <mailto:paul@grammatech.com>>; Larry Golding (Myriad Consulting Inc) <v-lgold@microsoft.com <mailto:v-lgold@microsoft.com>>; Yekaterina O'Neil <katrina@microfocus.com <mailto:katrina@microfocus.com>>; sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> *Cc:* David Keaton <dmk@dmk.com <mailto:dmk@dmk.com>> *Subject:* RE: [sarif] RE: your thoughts on DARIF It seems like we've got the nucleus of an interesting moving forward plan. 1)Invoke the OASIS machinery to expand the charter of this technical committee to include dynamic analysis results/metrics. 2)As part of this process, remap the 'S' in SARIF to 'Software' to retain the brand. Bless the new proposed standard as a v3 effort to indicate it is a major revision. 3)Explore moving forward funding/staffing of the technical committee. David pointed out offline that convening a moving forward v3 effort might also create more flexibility in bug-fixing the v2 spec along the way (a point I agree with). I will float this idea around MS a bit. On a separate note, a different team approached me on producing a standard that attempts to codify a complete data model for end-to-end engineering, including static analysis. Think models to express projects, people, release pipelines, deployment environments, analysis results, etc. etc. and linking them together. For quality, compliance, auditing, chain-of-custody purposes. Michael *From:*Paul Anderson <paul@grammatech.com <mailto:paul@grammatech.com>> *Sent:* Friday, July 19, 2019 9:24 AM *To:* Larry Golding (Myriad Consulting Inc) <v-lgold@microsoft.com <mailto:v-lgold@microsoft.com>>; Michael Fanning <Michael.Fanning@microsoft.com <mailto:Michael.Fanning@microsoft.com>>; Yekaterina O'Neil <katrina@microfocus.com <mailto:katrina@microfocus.com>>; sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> *Subject:* Re: [sarif] RE: your thoughts on DARIF All: We're very interested too. We've had some success with expressing dynamic properties in SARIF already. I would argue that we should work to have one standard that can satisfy both needs. Perhaps the 'S' should stand for just 'Software'. Having said that, someone here is also looking at using it to express properties in HDLs. To express test coverage results it will be important to express metrics in the format. For example, a commonly used metric is the Test Effectiveness Ratio (TER), which is percentage of the function that got covered by the test suite. It would be super useful to be able to express that in SARIF. (Some organizations have rules that require TER to be greater than some threshold, and violations of that rule can then be expressed as SARIF results.) Many other dynamic results (e.g., profiling) are also expressed as metrics, so I believe that's the biggest gap in SARIF for dynamic use. -Paul On 7/18/2019 6:53 PM, Larry Golding (Myriad Consulting Inc) wrote: Yes, I would also be very interested in exploring this. Larry *From:* sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> <sarif@lists.oasis-open.org> <mailto:sarif@lists.oasis-open.org> *On Behalf Of *Michael Fanning *Sent:* Thursday, July 18, 2019 3:51 PM *To:* Yekaterina O'Neil <katrina@microfocus.com> <mailto:katrina@microfocus.com>; sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> *Subject:* [sarif] RE: your thoughts on DARIF Microsoft would definitely be interested in exploring this possibility. I was just pinged this week on the possibilities using SARIF for capturing code coverage results (which are apparently of significant compliance interest for the automotive industry). This would be a good 'seed' scenario for a dynamic analysis initiative (along with profiling, web testing, etc.). Pull us in when you're ready to talk. I will help connect the conversation with MS teams who are stakeholders in these areas.https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FISO_26262&data=02%7C01%7Cv-lgold%40microsoft .com%7C17889714dc9d410bae0a08d7107b5f8b%7C72f988bf86f141af91ab2d7cd011 db47%7C1%7C0%7C636995994388361434&sdata=zibif1kJUvPcypg5DLjAQ1H36o ucoXz%2BchhucUoQyvI%3D&reserved=0<https://en.wikipedia.org%2Fwiki%2FISO_26262&data=02%7C01%7Cv-lgold%40microsoft.co m%7Cfa961fa69a604b9f9ed008d70ca1ad4e%7C72f988bf86f141af91ab2d7cd011db4 7%7C1%7C0%7C636991760851525139&sdata=gfCMFDwJsFCodIaepYRapT%2BSGUS1RmS BlwSUFjSFRyY%3D&reserved=0> *From:* sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> <sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org>> *On Behalf Of *Yekaterina O'Neil *Sent:* Thursday, July 18, 2019 11:16 AM *To:* sarif@lists.oasis-open.org <mailto:sarif@lists.oasis-open.org> *Subject:* [sarif] your thoughts on DARIF Hi all, As we are hopefully approaching the release of the first version of the SARIF standard, I wanted to bring up the conversation we had earlier about standardizing dynamic results. I was curious who on this list would potentially be interested in pursuing a DARIF some time in the future. Thanks! k -- Paul Anderson, VP of Engineering, GrammaTech, Inc. 531 Esty St., Ithaca, NY 14850 Tel: +1 607 273-7340 x118;https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2F www.grammatech.com&data=02%7C01%7Cv-lgold%40microsoft.com%7C178897 14dc9d410bae0a08d7107b5f8b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0% 7C636995994388361434&sdata=KIPWqIGSNbGnrv1yB%2FiFXkHh3oDx5w6Dt4rIK EAnIWg%3D&reserved=0 <http://www. grammatech.com&data=02%7C01%7Cv-lgold%40microsoft.com%7Cfa961fa69a604b 9f9ed008d70ca1ad4e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636991 760851535137&sdata=IlFiKIhw6kBxw3MbUEVUDFGNQVaBQaHhT44tIplz4U0%3D&rese rved=0>--------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php&data=02%7C01%7Cv-lgold%40microsoft.com%7C17889714dc9d410bae0a08d7107b5f8b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636995994388361434&sdata=E4mujtlrtvOAOuciV3pc2cGCt4sfjeZWq2qFfLwG1ac%3D&reserved=0 --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php&data=02%7C01%7Cv-lgold%40microsoft.com%7C17889714dc9d410bae0a08d7107b5f8b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636995994388361434&sdata=E4mujtlrtvOAOuciV3pc2cGCt4sfjeZWq2qFfLwG1ac%3D&reserved=0 --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
-- Paul Anderson, VP of Engineering, GrammaTech, Inc. 531 Esty St., Ithaca, NY 14850 Tel: +1 607 273-7340 x118; http://www.grammatech.com
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]