CleanSpeak 2.0 Overview
This chapter gives you a brief overview of the layout of the CleanSpeak system and what is provided to you. It also tells you things you might need to build and how the various integrations work.
This diagram gives you a general description of a the possible ways that CleanSpeak is setup and integrated with your applications. As you can see near the top are various applications that you want to integrate with CleanSpeak. These applications are using the CleanSpeak WebService for either filtering or moderation (depending on which products you need and have licensed). These applications communicate with the WebService via a REST API over HTTP.
The WebService loads and stores data from a relational database using SQL. This database is setup and configured by following the instructions in the Installation section of these docs. If you are using the Real-Time Filter product, the WebService will load the white and black lists it uses to filter from the database. If you are using the Moderator product, the items you send to the WebService to be moderated are stored in the database.
The Management Interface is the web application that is used to update the white and black lists used by the Real-Time Filter as well as to moderate content sent to the Moderator. This interface runs separately from the WebService and communicates with the database to perform these tasks.
You’ll notice that both the Management Interface and the WebService use the same database. This allows the Management Interface to make changes to the white and black lists and then for the WebService to query the database for those changes.
Another component in the diagram is the Search Engine. This component is used in conjunction with the Moderator product to provide full text searching of content.
The last component of this diagram is the Moderation Notification WebService. This component is something you will need to write if you intend to use the Approval Moderation Queue features of CleanSpeak. The requirements for this component are covered in this documentation.
Next Steps
In the next chapters we will cover all of the system and software requirements for CleanSpeak, detailed installation instructions, configuration, how to run CleanSpeak, and how to use CleanSpeak.
Important Notices
Version 2.0
This is a major release of the CleanSpeak platform and a lot of changes have been made to the system, APIs and the database. We have worked hard to test the migration scripts and update process, but nothing is ever perfect. If you want to upgrade to 2.0 but would like assistance, please contact Inversoft Support to discuss our services.
Requirements
There are a number of requirements to use CleanSpeak. There are also a number of technologies that are required depending on how you want to use CleanSpeak. Installation of all of these components (except the operating system) are covered in the Installation chapter. The required technologies are:
Operating System
Clean Speak is capable of running on most modern operating systems. Below is a list of the supported operating systems.
- Fedora 10 or higher
- Redhat ES 5 or higher
- CentOS 5.2 or higher
- Debian 5 or higher
- Ubuntu 8 or higher
- Windows XP or higher
- Windows Server 2003 or higher
- Solaris 10 or higher
- Open Solaris 2008.05 or higher
- Mac OS X 10.5.4 or higher (64-bit Intel only)
- Mac OS X-Server 10.5.4 or higher (64-bit Intel only)
Many other operating systems are capable of running the CleanSpeak, but are not officially supported. Any operating system capable of running Java version 1.6 or higher should be capable of running CleanSpeak.
Java
You must have Java installed to use CleanSpeak. Here are a list of the acceptable versions of Java.
- JDK 6.0 or higher from Sun
- OpenJDK 6.0 or higher
- JDK 6.0 or higher from Apple
JEE Server for WebService and Management Interface
In order to use the Management Interface or WebService, you must have a JEE server installed. The officially supported JEE server is:
- Bundled Tomcat version 7.0.16
The CleanSpeak Management Interface and WebService are capable of running on any JEE-compliant web server. Inversoft only officially supports the bundled Tomcat releases. If you are interested in using another JEE web server, Inversoft recommends Oracle’s WebLogic or Sun’s GlassFish. However, we do not support these web servers.
NOTE: Versions of Tomcat that are available for Red Hat and Debian systems via Yum or Aptitude are not valid Tomcat releases. Both of these distributions have made changes to Tomcat that are non-standard. For this reason, Inversoft does not support these versions of Tomcat.
Database
There are a couple of different options for the database setup to use with CleanSpeak. Below are the types of databases that are currently supported.
- No database
- MySQL 5.0 or higher
- PostgreSQL 9.1 or higher
If you don’t intend to use the Management Interface and want to only use the CleanSpeak Real-Time Filter using the XML database file and not a relational database then you don’t need a relational database installed. If you are using the Management Interface, Moderator, or Real-Time Filter using a relational database, then you will need one of the supported relational databases installed.
File Systems
If you are running the CleanSpeak Moderator and have content storage and searching enabled, you will need to deploy the CleanSpeak Search Engine application to a server with a fast and large enough hard-disk drive to accommodate all of your content. We suggest using a minimum of SATA 7,200 RPM drives in a RAID 1 or higher configuration.
Upgrading from 1.x
If you are upgrading from any version 1.x you will need to take a number of steps to complete the upgrade. These steps are:
- Pending Approvals: First, login to your existing Management Interface and check to see if there are any pending approvals System -> Approvals. Approve or reject any that are pending. There must be no pending approvals prior to the upgrade.
- Database Migration: The core change related to filtering with CleanSpeak 2.0 is that the concept of Contexts has been removed and instead of having one Category, each blacklist entry can now have multiple Tags. Your existing Contexts and Categories will both be migrated as Tags for each entry.
Check the version you are currently running. You must be at 1.3.x in order to migrate to 2.0. If you are running 1.0, 1.1, or 1.2, you need to run the respective migration scripts to get to 1.3 first. All update scripts are available in your account from Downloads -> Upgrade Downloads.
PostgreSQL Users: You will need to ensure that the user you run the update as has permissions to create store procedures and also has permissions to load C based stored procedures from disk. You also need to ensure that you have the UUID-OSSP library installed in your PostgreSQL installation because it is loaded as part of the upgrade process. To give the cleanspeak user these permissions, you can grant the SUPERUSER role to it and then later revoke it. The steps to grant this role are:
$ psql -Upostgres cleanspeak postgres> alter role cleanspeak SUPERUSER
- Software Update: CleanSpeak 2.0 downloads are available from your account. If you will be using the Moderation Tools, you will need to install the Search Engine after updating the Management Interface and Webservice.
- Tomcat Updates: You must reconfigure the Tomcat server.xml using the server.xml-example files in the distributions. This needs to be done for the Management Interface and WebService. We updated to Tomcat 7.0 and this new version of Tomcat uses different server.xml configuration parameters.
- Integration Code: CleanSpeak 1.0 implemented an XML API and later introduced a REST API with version 1.2. The XML API has been completely removed with 2.0 and the REST API has been changed. Review the WebServices section and specifically Content Items to update your filtering integration code.
- 1.x Modertor: If you are using the 1.x Moderator product, contact us for details with the upgrade.
- License File: Your 1.x license file will need to be updated. Login to your account and click Generate 2.x License File and review the license file installation doc.
Upgrading from 2.0 Pre-Releases
The migration from any pre-release of 2.0 (2.0-Bx or 2.0-RCx) to the full version of 2.0 requires a number of modifications to work properly. These changes are outlined below.
1. Update your license file to a 2.0 license
You can download a 2.0 license file from the Inversoft website and install it as normal.
2. Change your API calls to the new API URLs and parameters
To increase performance and provide consistency across all our APIs, we have made a number of minor changes to the APIs. The make the update as simple as possible, use this follow instructions to update your application code to use the new APIs.
Changes to /content/handle
- /content/handle has been moved to /content/item
- The names of the request parameters for the /content/item request have changed. Rather than prefixing everything with “request.” we have changed the request parameters to indicate their use. Use the table below to identify the old parameter name and the new parameter name. NOTE: the values you need to pass in the request have NOT changed
- The request parameters for the /content/user request have changed. We have standardize all of these parameters to be prefixed by “contentUser.”. Use the table below to identify the old parameter name and the new parameter name
The changes to the request parameters from /content/handle to /content/item are listed in this table:
| Old Parameter | New Parameter |
|---|---|
| request.content | contentItem.content |
| request.contentType | contentItem.type |
| request.application | contentItem.application |
| request.component | contentItem.component |
| request.moderation.createDate | contentItem.createInstant |
| request.moderation.locationUID | contentItem.location |
| request.moderation.senderUID | contentItem.senderUID |
| request.moderation.receiverUID | contentItem.receiverUID |
| request.moderation.clientUID | contentItem.uid |
| request.moderation.type | moderation.type |
| request.filter.operation | filter.operation |
| request.filter.cleanspeakdb.enabled | filter.blacklist.enabled |
| request.filter.cleanspeakdb.tag | filter.blacklist.tag |
| request.filter.cleanspeakdb.locale | filter.blacklist.locale |
| request.filter.cleanspeakdb.severity | filter.blacklist.severity |
| request.filter.characters.enabled | filter.characters.enabled |
| request.filter.characters.character | filter.characters.character |
| request.filter.emails.enabled | filter.emails.enabled |
| request.filter.emails.maximumMatchLength | filter.emails.maximumMatchLength |
| request.filter.emails.spacePenalty | filter.emails.spacePenalty |
| request.filter.emails.domainQuality.domain | filter.emails.domainQuality.domain |
| request.filter.emails.domainQuality.quality | filter.emails.domainQuality.quality |
| request.filter.phoneNumbers.enabled | filter.phoneNumbers.enabled |
| request.filter.phoneNumbers.maximumMatchLength | filter.phoneNumbers.maximumMatchLength |
| request.filter.phoneNumbers.minimumMatchLength | filter.phoneNumbers.minimumMatchLength |
| request.filter.phoneNumbers.spacePenalty | filter.phoneNumbers.spacePenalty |
| request.filter.phoneNumbers.separatorPenalty | filter.phoneNumbers.separatorPenalty |
| request.filter.phoneNumbers.wordPenalty | filter.phoneNumbers.wordPenalty |
| request.filter.urls.enabled | filter.urls.enabled |
| request.filter.urls.maximumMatchLength | filter.urls.maximumMatchLength |
| request.filter.urls.spacePenalty | filter.urls.spacePenalty |
| request.filter.urls.domainQuality.domain | filter.urls.domainQuality.domain |
| request.filter.urls.domainQuality.quality | filter.urls.domainQuality.quality |
| request.filter.words.enabled | filter.words.enabled |
| request.filter.words.word | filter.words.word |
Changes to /content/user
- The request parameters for the /content/user request have changed names and in some cases the values you need to send in are different
- The HTTP methods used to call the /content/user have changed. No longer is the POST method used for creates and updates. Instead, the POST method now only performs a create while the PUT method is used for updates.
The changes to the request parameters for /content/user are listed in this table:
| Old Parameter | New Parameter |
|---|---|
| uid | contentUser.uid |
| name | contentUser.name |
| contentUser.email | |
| birthDate | contentUser.birthDateInstant |
| lastLogin | contentUser.lastLoginInstant |
| attributes['name'] | contentUser.attributes['name'] |