long-term investment that goes beyond
the application itself. Therefore, it’s
important to select a vendor that aligns
with the values and principles of the enterprise—for example, software design
discipline (scale and performance),
data security and privacy management,
use of open standards, and ease of operations and maintenance.
To ensure a vendor meets its requirements, an enterprise needs a rigorous evaluation and validation process. Two distinct sets of evaluations
determine and shape the reliability of
˲ Functional evaluation: represents
the business functionality required by
˲Infrastructure evaluation: represents the application’s IT requirements.
Functional evaluation. Functional
requirements are derived directly
from customer objectives and form
the basis of the evaluation process.
Each functional requirement has a set
of key functional characteristics. The
goal of the evaluation process is to do
an in-depth analysis of these characteristics and assess the feasibility of
To understand this, consider the
following scenario. Assume your enterprise is evaluating a third-party IT
inventory system to manage your corporate IT asset information. One of
your business objectives is to predict
the supply and demand for your inventory in realtime. This could result in a
requirement for a centralized global
inventory database that updates in realtime every time a checkout happens.
Based on this scenario, let’s analyze
the core characteristics that a functional evaluation should delve into.
Functional specification. Does the
vendor understand the functional requirement and the expected outcome?
In the scenario just described, the
functional requirement is to maintain
a global inventory database for all asset
information. The expected outcome is
the ability to track asset information
and update the global inventory database in realtime.
Dependencies and constraints. Does
the vendor need to be aware of any core
dependencies or constraints? For ex-
ample, does the global inventory data-
base depend on any external entities?
Is a centralized database required for
reads and writes, or is a distributed
setup required? What are the pros and
cons of both approaches?
Functional interfaces. Does the ven-
dor understand all the end-to-end
functional interfaces involved in this
requirement? For example, does the
inventory database have any reporting
interfaces? How does the admin in-
teract with the database? How do the
users interact with the database when
they do a checkout? What is the end-to-
Geographic requirements. Does the
enterprise have a presence across the
globe? Will users access this inventory
system from different regions? What
are the specific performance and laten-
cy requirements for these users?
Scale and load requirements. How
many users are going to use the invento-
ry system, both globally and per region?
What are the QPS (queries per second)
or load requirements for these users?
Are there any peak or off-peak volume
requirements or considerations?
Security requirements. Does the
vendor understand the security pos-
ture of the system? Are there any spe-
cific access restrictions based on user
type (for example, admin vs. normal
user)? What is the authentication and
authorization mechanism? Does the
application depend on a centralized
authorization service such as LDAP
(Lightweight Directory Access Proto-
col) or AD (Active Directory)? Is there a
single sign-on dependency?
Compliance requirements. Does the
vendor understand and meet the compli-
ance requirements for this application?
Handling requirements. Does the
vendor understand the key failure
modes based on the design of the sys-
tem? How does the vendor’s software
handle exceptions (for example, re-
quest timeouts, retries during write
failures, and connection resets)?
Release management. What software
release management discipline does
the vendor use? What is the release
cycle? How are changes tested before
being released to the customer? What
is the QA/qualification process?
Testing and validation. Does the ven-
dor have a holistic testing plan that
covers the end-to-end workflow, and
does it include all the edge cases? What
is the testing plan for measuring load
early on leads to a
better design and
the overall system.
Arriving at a clear
set of measurable
SLOs, however, is an