How to Select a SysML Modeling Tool for MBSE

Selecting the best Systems Modeling Language (SysML) tool for your Model-Based Systems Engineering (MBSE) project can be difficult, even if you are a skilled visual modeler and an expert Systems Engineer. SysML is a powerful, but formidable, technical language to master, and the “Muddle-Driven Marketecture” vendor hype and tool featuritis associated with commercial SysML tools can overwhelm even savvy engineers. Systems Engineers who want to transition to MBSE technologies need to climb two steep learning curves to reap their benefits:

1. achieving fluency in SysML 1.x as a lingua franca for technical communication; and
2. learning how to drive the “clickology” of a SysML modeling tool, many of which have mediocre (or worse) user interfaces.
   

Background: The Importance of Tool Selection Due Diligence

Since Model-Based Systems Engineering project stakes are high in terms of both time and budget, it behooves you and your team to choose your SysML modeling tool wisely. While the efficient use of your MBSE team’s time is of paramount importance, it is also noteworthy that the prices of commercial SysML modeling tools vary by more than an order-of-magnitude (\(\mathcal{O}(10)\)). The prices for SysML commercial modeling tools with comparable features range from \(\begin{align}\sim\$\end{align}\) 300 to \(\begin{align}\$\end{align}\) 10K+, which is greater than an \(\mathcal{O}(10)\) difference, where modeling tool quality is not always directly proportional to price. (Indeed, a case can be made for inverse proportionality between tool quality and price—see Nyrbok’s Laws of Software.) Since many Systems Engineers will also need to consider the associated training and coaching (consulting) costs for the SysML tool they choose, keep in mind that more expensive tools can also be more difficult to drive. So do you want to blow your SysML tool and training budget on a \(\begin{align}\$\end{align}\) 10K/seat SysML modeling tool that drives like a rough-handling semi-trailer truck, or a \(\begin{align}\$\end{align}\) 300 SysML tool that drives like a nimble sports car? For a 10+ person MBSE team with tough deadlines and a tight budget, you likely need to make a smart tool choice.

\(\begin{align} U & \propto \frac{k}{{F \times C}}, \\ P & \propto {F \times S}, \\ Q & \propto \frac{U}{P}, \\\text{where}~F &=\text{number of Tool Features,}\\ C &=\text{average #Clicks per Feature},\\ S &=\text{Sales & Marketing budget},\\ U &=\text{Usability}, \\ P &=\text{Price}, \\ Q &=\text{Quality}. \\ \end{align}\)

— Nyrbok’s Laws of Software: Usability, Pricing, and Quality
For example, consider a 20-person MBSE project with a \(\begin{align}\$\end{align}\) 200K budget for SysML modeling tools and training. Will the project team be better off spending \(\begin{align}\$\end{align}\) 200K for 20 licenses at \(\begin{align}\$\end{align}\) 10K/seat with \(\begin{align}\$\end{align}\) 0K left over for basic training, or spending \(\begin{align}\$\end{align}\) 6K for 20 licenses at \(\begin{align}\$\end{align}\) 300/seat with \(\begin{align}\$\end{align}\) 194K left over for advanced training and expert coaching? The answer, of course, depends upon many factors other than price. In addition to the advanced features frequently hyped by high-cost SysML tool vendors to justify their prices (notably model simulation and executability capabilities), you also need to consider more mundane, but essential features, such as usability (a.k.a. ease-of-use) and standards compliance. After all, what good is a pricey executable SysML modeling tool if its user interface is so counter-intuitive that your Systems Engineers revert to Visio drawings + Excel spreadsheets for their project deadline deliverables? (This may help explain why the Visio drawing tool remains our planet’s most popular “modeling tool.”)
Given the importance of SysML modeling tool selection for your Model-Based Systems Engineering, we recommend that medium-to-large size projects (10+ software developers) conduct a bona fide trade study for their SysML tool selection process. For smaller projects with smaller teams (< 10 Systems Engineers), you may want a less formal and more “Agile” (as in Agile Development and Agile Modeling) tool selection process. In any case, you need to perform some basic due diligence for SysML modeling tool selection, lest you squander your project budget and blow your delivery schedule. (Caveat emptor applies here, as elsewhere!) Here are the basic steps that your SysML modeling tool selection process should follow:

1. Specify objectives and requirements
2. Define selection criteria
3. Assign relative weights to selection criteria
4. Identify candidate SysML tools
5. Evaluate candidate SysML tools
6. Select SysML tool

Specify objectives and requirements for your SysML tool

You should begin by specifying clear and precise objectives and requirements for your SysML modeling tool. Have you defined a pragmatic objective for your SysML tool consistent with the practical goals of your MBSE project? An example of a pragmatic and measurable objective is: “The project’s SysML modeling tool will be able to specify the following Systems Engineering artifacts: Capability Objectives, Concept of Operations (CONOPS), Requirements Space, Performance Measures and Methods, Trade Studies, System Architecture, …”

You should also specify specific functional and non-functional requirements for your SysML tool. Your functional requirements should define the essential and advanced features of an acceptable SysML tool. An example of an essential functional requirement is: “The project’s SysML tool must be able to draw all nine diagrams and Allocation tables as specified and illustrated in the OMG SysML v. 1.x specification.” An example of an advanced functional requirement is: “The project’s SysML tool must be able to simulate all Trade Studies artifacts specified as Parametric diagrams, and include the ability to interactively change input/output parameters to facilitate hypothetical reasoning scenarios.” Your non-functional requirements should specify usability and performance requirements, along with metrics for measuring them. An example of a usability requirement is: “The project’s SysML tool User Interface (UI) must provide context sensitive help that explains all common modeling functions and how to achieve common modeling tasks.” An example of a performance requirement is: “The SysML tool’s team modeling capability must be able to support up to 100 total users and 50 concurrent users, while maintaining a UI response time limit of 1 second for common editing operations and a UI response time of 10 seconds for model repository check-out/check-in operations.”

Define evaluation criteria

After you have specified your requirements your should define objective evaluation criteria for determining which SysML tools satisfy those requirements. Determine a balanced set of tool characteristics that are consistent with your requirements, and define objective criteria for measuring those characteristics, including clear thresholds beyond which a candidate SysML tool is deemed unsatisfactory. For example, consider the following SysML modeling tool evaluation criteria, which are elaborated up in the related How to Define SysML Tool Evaluation Criteria article: Usability (Ease-of-Use), Functionality (Drawing, Simulation & Execution), Standards Compliance, Interoperability, Technical Support, Team Modeling Support, and Value.

Assign relative weights to evaluation criteria

After you have specified your selection criteria you should assign numerical weights to them to quantify their relative importance. The assignment of relative numerical weights to your criteria allows you to tailor your tool selection to specific team and project needs, and it can also reduce evaluator bias. For example, consider weighting the criteria in the evaluation criteria example above as follows:

\[Score_ {tool} = \overline{x} = \sum_{i=1}^{n} w_i * c_i \]

where Tool Evaluation Score (\(Score_{tool}\)) is the weighted mean (\(\overline{x}\)) of the evaluation criteria.

Note that the same evaluation criteria e.g., (\(c_{1}\) = Usability, \(c_{2}\) = Features: Drawing, \(c_{3}\) = Features: Modeling & Simulation, \(c_{4}\) = Standards & Interoperability, \(c_{5}\) = Technical & Team Modeling Support, \(c_{6}\) =Value) can be weighted differently for different team and project needs. For example, a small Agile Modeling team supporting an Agile development method such as Scrum with no need to generate software code from its models might assign the following weights to the evaluation criteria: (\(w_{1}\) = 20%, \(w_{2}\) = 20%, \(w_{3}\) = 0%, \(w_{4}\) = 20%, \(w_{5}\) = 20%, \(w_{6}\) = 20%). In contrast, a large Model-Driven Development team seeking to generate production software code from its models might assign the following weights: (\(w_{1}\) = 20%, \(w_{2}\) = 15%, \(w_{3}\) = 25%, \(w_{4}\) = 15%, \(w_{5}\) = 10%, \(w_{6}\) = 15%).

Identify candidate SysML tools

You are now ready to identify candidate SysML modeling tools to thoughtfully evaluate. It’s important to realize that you are not expected to evaluate all the SysML tools in the universe whose vendors claim SysML standards compliance. Indeed you will soon realize, if you haven’t already, that since there is no SysML reference implementation or test suite for testing SysML standard compliance, it is relatively easy for tool vendors to take implementation shortcuts (e.g., not supporting Allocation Tables, substituting proprietary requirement diagrams for SysML Requirement diagrams) that may fail to meet your requirements. First you should compile a “long list” of SysML candidate tools by starting with those that you may already know about or have been recommended to you, and extend it via professional networking and Google research. Then apply the SysML tool requirements that you have previously applied to quickly filter out non-contenders. For example, if your tool requirements include “… shall automatically generate documentation in HTML and RTF formats …” and a candidate tool does not include support for this function, it should immediately be dropped from your list. In most cases, if you have done a good job specifying your requirements, your “long list” should reduce into a manageable “short list” of five or fewer candidate SysML tools.

Evaluate candidate SysML tools

Now you can take your “short list” of candidate SysML tools and perform a hands-on evaluation of each of them, systematically applying your weighted evaluation criteria. In order to perform this evaluation fairly and consistently across competitive modeling tools, use the same SysML example model for test driving each tool. The SysML example model that you choose should be of moderate complexity (say 100+ Blocks with Properties and Operations, plus Activities/Sequences that use the Blocks for Swimlanes/Lifelines and Input/Output parameters). Your SysML example model should thoroughly exercise all the SysML diagram types defined in the latest SysML specification. If you don’t have your own SysML moderate-complexity example yet, use the examples in one of the better SysML text books as as a neutral starting point.

Select SysML tool

If you have followed the preceding steps carefully to this point, you will likely end up with two or three SysML tools that are fairly close in their cumulative evaluation ratings. No worries; this is to be expected. At this point, since it’s likely you’ve learned a lot about SysML modeling tools by your evaluation process to date, you should review your requirements and weighted evaluation criteria, and tune them based on what you have learned. In most cases, this will suffice to make a final SysML modeling tool selection.