Wednesday, January 27, 2016

apiman 1.2.1 Export and Import

The Question you Dread

If you use a computer at home or at work, you'll eventually find yourself in a situation where you lose some important data and, while you are trying to recover it, someone asks you a question that is simultaneously annoying and terrifying:

"Did you make a backup?"

Happily, the 1.2 release of apiman includes a new feature that enables you to export and import your apiman data and provides you with an easy means to create apiman data backups.  In this post,  we'll take a look at the new export/import feature, and how you can use it for a variety of tasks to protect your data, make your life easier, and enable you to avoid annoying and terrifying questions.

For test data, we'll use the same types of data (organizations, users, plans, policies, APIs and client apps), that we created in the initial post in this series. (  https://dzone.com/articles/impatient-new-users )

Note that since that post was written in January 2015, some of the names of the data elements have changed. You can either create the test data referred to in this post yourself, or you can import the data file attached to the post.

Export/Import in apiman 1.2.1

The new export/import feature in apiman 1.2.1 enables you to export your apiman configuration data to a file, which can later be imported into an apiman system. Note that this feature follows an all-or-nothing approach in that is does not support incremental backup and restore of selected apiman data.
The three main use cases that the export/import feature supports are:
  • Backing up all your apiman data
  • Upgrading to a newer version of apiman
  • Migrating apiman data from a Test environment to a Production environment
The export/import operations are only available to Administrative users. The export/import feature is accessed through the admin operations menu:

Image title

Once you select the Export/Import Data feature, this menu is displayed:

Image title

One thing to keep in mind is that while you are importing or exporting data, no changes to data should be made or else the export/import may encounter an error, or may result in incomplete results. To be safe, you should disable user access to the API Manager, both the Management UI and its REST interface, for the duration of the import or export operation.

We'll look at exporting data first.

Backing up apiman Data

To make a backup of all your apiman data, simply select the "Export All" button. The apiman data will be written to a file and downloaded by your browser. Your browser's settings will determine where the file is saved.

The apiman data is written to a file named: api-manager-export.json

As its name indicates, the apiman data is written in JSON form. This format provides us with  several advantages. First of all, it's the format in which apiman is able to import data. (We'll perform an import later in this post.) Secondly, it's a text file where the contents of the file are human readable. The content of the file is ALL the apiman data, both the data elements that you have created and the data elements with which apiman is preconfigured.
Reading this file can greatly increase your understanding of the elements that are defined in apiman. Let's take a look at the elements in the file:
  • Users - The preconfigured "admin" user is defined here, as are the new users we create.
  • Gateways - The preconfigured apiman Gateway is defined here.
  • Roles - The preconfigured, permission-based roles, and new roles that we create, are defined here. For example, the "OrganizationOwner" role is shown to have these permissions: [ "apiAdmin", "orgAdmin", "apiView", "orgEdit", "clientEdit", "clientAdmin", "planView", "orgView", "planAdmin", "clientView", "planEdit", "apiEdit" ]
  • Policy Definitions -  Next, the preconfigured policies, and new policies that we create, are defined here. For example: the "RateLimitingPolicy" is described as "Enforces rate configurable request rate limits on an API.  This ensures that consumers can't overload an API with too many requests."
  • The remainder of the file includes the elements that we create: Organizations, Plans, APIs, and Client Apps. For example, here is the definition of the "echo" API that we created:
 "Apis" : [ {  
  "ApiBean" : {  
  "id" : "echo",  
  "name" : "echo",  
  "description" : "The echo API",  
  "createdBy" : "serprov",  
  "createdOn" : 1453773184836,  
  "numPublished" : 1  
  },  
  "Versions" : [ {  
  "ApiVersionBean" : {  
   "id" : 10,  
   "status" : "Published",  
   "endpoint" : "http://localhost:8080/apiman-echo",  
   "endpointType" : "rest",  
   "endpointContentType" : "json",  
   "endpointProperties" : { },  
   "gateways" : [ {  
    "gatewayId" : "TheGateway"  
   } ],  
   "publicAPI" : false,  
   "plans" : [ {  
    "planId" : "gold",  
    "version" : "1.0"  
   } ],  
   "version" : "1.0",  
   "createdBy" : "serprov",  
   "createdOn" : 1453773184845,  
   "modifiedBy" : "serprov",  
   "modifiedOn" : 1453773312563,  
   "publishedOn" : 1453773327835  
  },  
  "Policies" : [ {  
   "id" : 14,  
   "type" : "Api",  
   "organizationId" : "ACMEServices",  
   "entityId" : "echo",  
   "entityVersion" : "1.0",  
   "name" : "BASIC Authentication Policy",  
   "configuration" : "{\"realm\":\"Echo\",\"requireBasicAuth\":false,\"staticIdentity\":{\"identities\":[{\"username\":\"user1\",\"password\":\"admin123!\"}]}}",  
   "createdBy" : "serprov",  
   "createdOn" : 1453773312553,  
    "modifiedBy" : "serprov",  
    "modifiedOn" : 1453773312553,  
    "definition" : {  
    "id" : "BASICAuthenticationPolicy",  
    "templates" : [ ],  
    "deleted" : false  
   },  
   "orderIndex" : 1  
  } ]  


One thing to remember is that the exported data file represents ALL apiman data. It's not yet possible to perform incremental data backups in apiman. If you attempt to import the data from this file into the same apiman installation from which it was generated, you will see unique primary key violations as the import operation will attempt to create duplicate data elements.

OK, now that we have this exported data file, what can we do with it?

Well, obviously, if something goes wrong with your installation of apiman, you can start over with a clean installation, and instead of manually recreating your data, you can import the data. (Personal note from the author: I work in software test/QE. Part of our testing is always destructive in nature. As a result, we are always "messing up" test data. The export/import feature enables us to quickly reinstall apiman and recover a clean test environment.) To perform the import after a new installation of apiman, you simply select and upload  the exported data file:

Image title

The Management UI displays the status of the import as the data is processed:

Image title

How else can we use the exported data file?

Upgrading to a Newer Version of apiman

One of the best aspects of open source projects is the rapid rate at which new features are implemented and new versions are released. It's exciting to watch projects quickly mature as features are added and bugs are fixed, and since the projects are open source, you can even make your own contributions. This has been the case with apiman over the past several months. New features have been added such as metrics and support for creating custom policies.

One downside to all the rapid change is that as new versions of apiman have been released, we've had to recreate all our apiman data as there was no way to migrate apiman data from one release to the next. The export/import feature now gives us a way to export apiman data from one apiman release and import it into a new apiman release.

NOTE: In cases where the apiman data model changes between versions, apiman will introduce tools to transform the JSON export file from an older format to the latest.  It has not yet been decided whether those tools will be built into the Import process, or released as a standalone utility.

Migrating apiman Data from a Test Environment to a Production Environment

It's a common practice for organizations to maintain two separate installations of software releases:
  • A test environment, where the goal is to experiment with new features. This is typically an internal environment that gives up some measure of stability in exchange for the ability to "try out" new features as they become available. The rate of change for this environment is high as any disruption in service in this environment do not affect customers.
  • A production environment, where the goal is stability. This is the environment that supports your customers. Changes happen slowly in this environment and new features are only installed after they are carefully tested as disruptions in service in this environment do affect customers.
The new export/import feature in apiman 1.2 makes it possible for you to experiment with changes in your test environment, and, after the changes have been found to be stable, to easily migrate your test data from the test environment into your production environment. (You will, of course, create a backup of your production environment data before making any changes.  ;-)

Migrating apiman from one storage solution to another

Finally, when apiman is first installed, you must make various decisions about its configuration.  One of these decisions is where to store configuration and data.  When you first install apiman, you might decide that MySQL is the right choice.  However, somewhere along the line you might change your mind - perhaps you want to switch to postgresql, or even more drastically you might switch to Elasticsearch!  The Export/Import process described here solves the problem of how to migrate all your data from one storage location to another.

The process is basically the same as upgrading to a newer version of apiman.  But instead you will be upgrading to the same version of apiman, but with a different configuration.  Because the exported data is in a neutral (JSON) format, we can easily import into the new configuration.  This will result in all your data being migrated from MySQL to Elasticsearch (for example).

In Conclusion

The new export/import feature in apiman 1.2 provides an easy way to safeguard your apiman data and to make it possible to migrate your data between apiman releases and installations. In addition, since the exported data is human readable, it is a great resource for better understanding apiman data structures. And, it's easy to use too!

Cross-posted to:

Monday, August 17, 2015

Rate Based Policies and Quotas in JBoss apiman






In this, the sixth article in the series on apiman, JBoss’ new API Management framework, we’ll examine how apiman enables you to govern access to managed services through the use of rate limiting policies.



The runtime core of apiman is the API Gateway and the policies that it applies to incoming requests to services. apiman is configured out of the box with a variety of policies that can be used to govern access to services managed by the API Gateway based on IP address, user authentication, and usage levels. From its first release, apiman has supported rate limiting policies, where the upper limit for use of a service could be governed by a policy. In its new 1.1.6 release, apiman has expanded this support to include quota based limiting policies.


Types of Limiting Policies


As of release 1.1.6, apiman supports these types of limiting policies:


  • Rate Limiting - This policy type governs the number of times requests are made to a service within a specified time period. The requests can be filtered by user, application, or service and can set the level of granularity for the time period to second, minute, hour, day, month, or year. The intended use of this policy type is for fine grained processing (e.g., 10 requests per second).
  • Quota - This policy type performs the same basic functionality as the Rate Limiting policy type., however, the intended use of this policy type is for less fine grained processing (e.g., 10,000 requests per month).
  • Transfer Quota - In contrast to the other policy types, Transfer Quota tracks the number of bytes transferred (either uploaded or downloaded) rather than the total number of requests made.


Each of these policies, if used singly, can be effective in throttling requests. apiman, however, adds an additional layer of flexibility to your use of these policy types by enabling you to use them in combinations. Let's look at a few examples.


Combinations of Limiting Policies = Flexibility


Limiting the total number of service requests within a period of time, is a straightforward task as this can be configured in a quota policy. This policy, however, may not have the desired effect as the quota may be reached early in the defined time period. If this happens, the requests made to the service during the remainder of the time period will be blocked by the policy. A better way to deal with a situation like this is to implement a more flexible approach where the monthly quota policy is combined with a fine grained rate limiting policy that will act as a throttle on the traffic.


To illustrate, there are about 2.5 million seconds in a month. If we want to set the service request quota for a month to .5 million, then we can also set a rate limit policy to a limit of 5 requests per second to ensure that service requests are throttled and the service can be accessed throughout the entire month.


Here’s a visual view of a rate limiting policy based on a time period of one week. If we define a weekly quota, there is no guarantee that users will not consume that quota before the week is over. This will result in a service requests being denied at the end of the week:



In contrast, if we augment the weekly quota with a more fine grained policy, we can maintain the service’s ability to respond to requests throughout the week:



The ability to throttle service requests based on service request counts and bytes transferred provides even greater flexibility in implementing policies. Services that transfer larger amounts of data, but rely on fewer service requests can have that data transfer throttled on a per byte basis. For example, a service that is data intensive, will return a large amount of data in response to each service request. The service may only receive a request a few hundreds of times a day, but each request may result in several megabytes of data being transferred. Let's say that we want to limit the amount of data transferred to 6GB per hour. For this type of service, we could set a rate limiting policy to allow for one request per minute, and then augment that policy with a transfer quota policy of 100Mb per hour.


Summary


When you configure limiting policies with apiman, it's important to remember that the limits you set can not only function as hard limits for service requests, they can also be used to throttle service request. This throttling gives you the flexibility to control the level of incoming service requests over a period of time that you designate in the policies without blocking all incoming service requests. The flexibility that apiman provides you in configuring limiting policies is further enhanced by its support for you to create combinations of limiting policies. These combined policies work together to give you both coarse grained and fine grained control over incoming service requests.  


Author Acknowledgements


As always, the author would like to acknowledge Eric Wittmann and the apiman team for their review comments and suggestions on writing this article!



Saturday, August 1, 2015

apiman Policy and Endpoint Security


Screenshot-6.png




In this, the fifth article in the series on apiman, JBoss’ new API Management framework, we’ll examine how apiman enables you to provide security for your managed services at the policy level, and and at the endpoint level for its managed and unmanaged endpoints.




Unintentionally Insecure?


If you read the first article in this series closely (https://dzone.com/articles/impatient-new-users) you might have noticed that in the course of creating a service whose endpoint was managed by the apiman API Gateway, we also inadvertently left that service in a very insecure state as unauthorized client applications could bypass the gateway and access the service directly.  We discussed how to configure authentication in a policy for the managed service endpoint in the most recent post in this series (https://dzone.com/articles/adding-basic-authentication). This authentication policy provides username/password security for clients as they access the managed service through the API Gateway, but it does not protect the service from unauthorized access attempts that bypass the Gateway completely. To make the service secure from unauthorized client applications, endpoint level security should also be configured.


In this article, we’ll examine both apiman policy level and endpoint level security, how they compare, and how they differ.


Complementary Types of Security Provided by apiman


The best way to start our discussion of the different, but complementary types of security that we’ll examine in this article is with a diagram. The nodes involved are the client applications that will access our services, the apiman API Gateway, and the servers that host our services:


Let’s work our way through the diagram from left to right and start by taking a look at Policy Level Security.


Policy Level Security


apiman includes several pre-defined policies OOTB. The policies provide support for controlling access to services based on the rate at which the services are invoked, the IP address of the client applications that access the services, authorization, and authentication. In the most recent article in this series, we showed how to configure an authentication policy. To keep things simple, we chose BASIC authentication. This BASIC Authentication policy provides security for the communication channel between the client applications and the apiman API Gateway. An incoming request to the API Gateway from a client initiates the policy chain, the policy is applied and the client is requested to supply a username and password.  The level of security provided by this policy can be enhanced if the policy is configured with SSL encryption.


But, this policy level security only secures the left side of the diagram, that is the communication channel between the applications and the API Gateway. In this communication channel, the applications play the role of the client, and the API Gateway plays the role of the server.


We also want to secure the right side of the diagram, where the API Gateway plays the role of a client, and the services play the role of the servers.


(It’s also worth noting that while policy security protects the managed service, it does nothing to protect the unmanaged service as this service can be reached directly, without going through the API Gateway. This is illustrated by the red line in the diagram. So, while access to the managed service through the apiman API Gateway is secure, policy security does not secure the unmanaged service endpoint.)


Endpoint Level Security


In contrast to policy level security, with endpoint security we are securing the right side of the diagram.


A recent post by Marc Savy to the apiman blog (http://www.apiman.io/blog/gateway/security/mutual-auth/ssl/mtls/2015/06/16/mtls-mutual-auth.html) described how to configure Mutually Authenticated TLS (Transport Layer Security) between the API Gateway and the managed services. With Mutual TLS, bi-direction authentication is configured so that the identities of both the client and server are verified before a connection can be made.


In setting up Mutual TLS, keystores, containing a node’s private key, and truststores, containing public certificates to govern the other nodes that the node should trust, were created. The API Gateway was configured in its apiman.properties file to reference the keystores and truststores.  The service was configured with mutual authentication by setting the API Security dropdown in the Implementation tab to MTLS/Two-Way-SSL. Finally, the service was programmed with mutual authentication enabled. With Mutual TLS configured, the communication channel on the right side of the diagram, from the API Gateway to the services, was made secure.


We should also note that, unlike policy security, endpoint security also secures the services from attempts to bypass the API Gateway. With Mutual TLS, a two-way trust pattern is created. The API Gateway trusts the services and the services trust the API Gateway. The services, however, do not trust the client applications. As is shown by the large “X” character that indicates that an application cannot bypass the API Gateway and access the services directly.


One last point that is important to remember is that the endpoint level of security applies to all requests made to the services, regardless of whatever policies are configured.


Compare and Contrast


To summarize, the differences between policy level security and endpoint level security are:


Policy Level Security
End Point Level Security
Secures communications between the applications (clients) and API Gateway (server)
Secures communications between the API Gateway (client) and services (servers)
Configured in an API Gateway policy
Configured for the API Gateway as a whole in apiman.properties and with key/certificates infrastructure
Applied by the policy at runtime
Applied for all service requests, regardless of the policies configured for a service
Does not secure the unmanaged service from access by unauthorized clients
Secures the unmanaged service endpoints from access by unauthorized clients

Author Acknowledgements


As always, the author would like to acknowledge Marc Savy, Eric Wittmann, and the apiman team for their review comments and suggestions on writing this article!


References