Running Istio Service Mesh on OpenShift

In the Kubernetes/OpenShift community everyone is talking about Istio service mesh, so I wanted to share my experience about the installation and running a sample microservice application with Istio on OpenShift 3.11 and 4.0. Service mesh on OpenShift is still at least a few month away from being available generally to run in production but this gives you the possibility to start testing and exploring Istio. I have found good documentation about installing Istio on OCP and OKD have a look for more information.

To install Istio on OpenShift 3.11 you need to apply the node and master prerequisites you see below; for OpenShift 4.0 and above you can skip these steps and go directly to the istio-operator installation:

sudo bash -c 'cat << EOF > /etc/origin/master/master-config.patch
admissionConfig:
  pluginConfig:
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission
    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission
EOF'
        
sudo cp -p /etc/origin/master/master-config.yaml /etc/origin/master/master-config.yaml.prepatch
sudo bash -c 'oc ex config patch /etc/origin/master/master-config.yaml.prepatch -p "$(cat /etc/origin/master/master-config.patch)" > /etc/origin/master/master-config.yaml'
sudo su -
master-restart api
master-restart controllers
exit       

sudo bash -c 'cat << EOF > /etc/sysctl.d/99-elasticsearch.conf 
vm.max_map_count = 262144
EOF'

sudo sysctl vm.max_map_count=262144

The Istio installation is straight forward by starting first to install the istio-operator:

oc new-project istio-operator
oc new-app -f https://raw.githubusercontent.com/Maistra/openshift-ansible/maistra-0.9/istio/istio_community_operator_template.yaml --param=OPENSHIFT_ISTIO_MASTER_PUBLIC_URL=<-master-public-hostname->

Verify the operator deployment:

oc logs -n istio-operator $(oc -n istio-operator get pods -l name=istio-operator --output=jsonpath={.items..metadata.name})

Once the operator is running we can start deploying Istio components by creating a custom resource:

cat << EOF >  ./istio-installation.yaml
apiVersion: "istio.openshift.com/v1alpha1"
kind: "Installation"
metadata:
  name: "istio-installation"
  namespace: istio-operator
EOF

oc create -n istio-operator -f ./istio-installation.yaml

Check and watch the Istio installation progress which might take a while to complete:

oc get pods -n istio-system -w

# The installation of the core components is finished when you see:
...
openshift-ansible-istio-installer-job-cnw72   0/1       Completed   0         4m

Afterwards, to finish off the Istio installation, we need to install the Kiali web console:

bash <(curl -L https://git.io/getLatestKialiOperator)
oc get route -n istio-system -l app=kiali

Verifying that all Istio components are running:

$ oc get pods -n istio-system
NAME                                          READY     STATUS      RESTARTS   AGE
elasticsearch-0                               1/1       Running     0          9m
grafana-74b5796d94-4ll5d                      1/1       Running     0          9m
istio-citadel-db879c7f8-kfxfk                 1/1       Running     0          11m
istio-egressgateway-6d78858d89-58lsd          1/1       Running     0          11m
istio-galley-6ff54d9586-8r7cl                 1/1       Running     0          11m
istio-ingressgateway-5dcf9fdf4b-4fjj5         1/1       Running     0          11m
istio-pilot-7ccf64f659-ghh7d                  2/2       Running     0          11m
istio-policy-6c86656499-v45zr                 2/2       Running     3          11m
istio-sidecar-injector-6f696b8495-8qqjt       1/1       Running     0          11m
istio-telemetry-686f78b66b-v7ljf              2/2       Running     3          11m
jaeger-agent-k4tpz                            1/1       Running     0          9m
jaeger-collector-64bc5678dd-wlknc             1/1       Running     0          9m
jaeger-query-776d4d754b-8z47d                 1/1       Running     0          9m
kiali-5fd946b855-7lw2h                        1/1       Running     0          2m
openshift-ansible-istio-installer-job-cnw72   0/1       Completed   0          13m
prometheus-75b849445c-l7rlr                   1/1       Running     0          11m

Let’s start to deploy the microservice application example by using the Google Hipster Shop, it contains multiple microservices which is great to test with Istio:

# Create new project
oc new-project hipster-shop

# Set permissions to allow Istio to deploy the Envoy-Proxy side-car container
oc adm policy add-scc-to-user anyuid -z default -n hipster-shop
oc adm policy add-scc-to-user privileged -z default -n hipster-shop

# Create Hipster Shop deployments and Istio services
oc create -f https://raw.githubusercontent.com/berndonline/openshift-ansible/master/examples/istio-hipster-shop.yml
oc create -f https://raw.githubusercontent.com/berndonline/openshift-ansible/master/examples/istio-manifest.yml

# Wait and check that all pods are running before creating the load generator
oc get pods -n hipster-shop -w

# Create load generator deployment
oc create -f https://raw.githubusercontent.com/berndonline/openshift-ansible/master/examples/istio-loadgenerator.yml

As you see below each pod has a sidecar container with the Istio Envoy proxy which handles pod traffic:

[centos@ip-172-26-1-167 ~]$ oc get pods
NAME                                     READY     STATUS    RESTARTS   AGE
adservice-7894dbfd8c-g4m9v               2/2       Running   0          49m
cartservice-758d66c648-79fj4             2/2       Running   4          49m
checkoutservice-7b9dc8b755-h2b2v         2/2       Running   0          49m
currencyservice-7b5c5f48fc-gtm9x         2/2       Running   0          49m
emailservice-79578566bb-jvwbw            2/2       Running   0          49m
frontend-6497c5f748-5fc4f                2/2       Running   0          49m
loadgenerator-764c5547fc-sw6mg           2/2       Running   0          40m
paymentservice-6b989d657c-klp4d          2/2       Running   0          49m
productcatalogservice-5bfbf4c77c-cw676   2/2       Running   0          49m
recommendationservice-c947d84b5-svbk8    2/2       Running   0          49m
redis-cart-79d84748cf-cvg86              2/2       Running   0          49m
shippingservice-6ccb7d8ff7-66v8m         2/2       Running   0          49m
[centos@ip-172-26-1-167 ~]$

The Kiali web console answers the question about what microservices are part of the service mesh and how are they connected which gives you a great level of detail about the traffic flows:

Detailed traffic flow view:

The Isito installation comes with Jaeger which is an open source tracing tool to monitor and troubleshoot transactions:

Enough about this, lets connect to our cool Hipster Shop and happy shopping:

Additionally there is another example, the Istio Bookinfo if you want to try something smaller and less complex:

oc new-project myproject

oc adm policy add-scc-to-user anyuid -z default -n myproject
oc adm policy add-scc-to-user privileged -z default -n myproject

oc apply -n myproject -f https://raw.githubusercontent.com/Maistra/bookinfo/master/bookinfo.yaml
oc apply -n myproject -f https://raw.githubusercontent.com/Maistra/bookinfo/master/bookinfo-gateway.yaml
export GATEWAY_URL=$(oc get route -n istio-system istio-ingressgateway -o jsonpath='{.spec.host}')
curl -o /dev/null -s -w "%{http_code}\n" http://$GATEWAY_URL/productpage

curl -o destination-rule-all.yaml https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/networking/destination-rule-all.yaml
oc apply -f destination-rule-all.yaml

curl -o destination-rule-all-mtls.yaml https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/networking/destination-rule-all-mtls.yaml
oc apply -f destination-rule-all-mtls.yaml

oc get destinationrules -o yaml

I hope this is a useful article for getting started with Istio service mesh on OpenShift.

2 Replies to “Running Istio Service Mesh on OpenShift”

  1. Hi, thanks for sharing the installer. I followed his tutorial and I get the following error in “istio-sidecar-injector-6f696b8495-n4cc2 0/1 CrashLoopBackOff 782 2d”
    ******************************************************************
    info Template: |
    rewriteAppHTTPProbe: false
    initContainers:
    [[ if ne (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) “NONE” ]]
    – name: istio-init
    image: “maistra/proxy-init-centos7:0.9.0”
    args:
    – “-p”
    – [[ .MeshConfig.ProxyListenPort ]]
    – “-u”
    – 1337
    – “-m”
    – [[ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode ]]
    – “-i”
    – “[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` “*” ]]”
    – “-x”
    – “[[ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` “” ]]”
    – “-b”
    – “[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` (includeInboundPorts .Spec.Containers) ]]”
    – “-d”
    – “[[ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` 15020 ) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` “” ) ]]”
    [[ if (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces`) -]]
    – “-k”
    – “[[ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` ]]”
    [[ end -]]
    imagePullPolicy: IfNotPresent
    resources:
    requests:
    cpu: 10m
    memory: 10Mi
    limits:
    cpu: 100m
    memory: 50Mi
    securityContext:
    capabilities:
    add:
    – NET_ADMIN
    privileged: true
    restartPolicy: Always
    [[ end -]]
    containers:
    – name: istio-proxy
    image: [[ annotation .ObjectMeta `sidecar.istio.io/proxyImage` “maistra/proxyv2-centos7:0.9.0” ]]
    ports:
    – containerPort: 15090
    protocol: TCP
    name: http-envoy-prom
    args:
    – proxy
    – sidecar
    – –domain
    – $(POD_NAMESPACE).svc.cluster.local
    – –configPath
    – [[ .ProxyConfig.ConfigPath ]]
    – –binaryPath
    – [[ .ProxyConfig.BinaryPath ]]
    – –serviceCluster
    [[ if ne “” (index .ObjectMeta.Labels “app”) -]]
    – [[ index .ObjectMeta.Labels “app” ]].$(POD_NAMESPACE)
    [[ else -]]
    – [[ valueOrDefault .DeploymentMeta.Name “istio-proxy” ]].[[ valueOrDefault .DeploymentMeta.Namespace “default” ]]
    [[ end -]]
    – –drainDuration
    – [[ formatDuration .ProxyConfig.DrainDuration ]]
    – –parentShutdownDuration
    – [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
    – –discoveryAddress
    – [[ annotation .ObjectMeta `sidecar.istio.io/discoveryAddress` .ProxyConfig.DiscoveryAddress ]]
    – –zipkinAddress
    – [[ .ProxyConfig.GetTracing.GetZipkin.GetAddress ]]
    – –connectTimeout
    – [[ formatDuration .ProxyConfig.ConnectTimeout ]]
    – –proxyAdminPort
    – [[ .ProxyConfig.ProxyAdminPort ]]
    [[ if gt .ProxyConfig.Concurrency 0 -]]
    – –concurrency
    – [[ .ProxyConfig.Concurrency ]]
    [[ end -]]
    – –controlPlaneAuthPolicy
    – [[ annotation .ObjectMeta `sidecar.istio.io/controlPlaneAuthPolicy` .ProxyConfig.ControlPlaneAuthPolicy ]]
    [[- if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port` 15020 ) “0”) ]]
    – –statusPort
    – [[ annotation .ObjectMeta `status.sidecar.istio.io/port` 15020 ]]
    – –applicationPorts
    – “[[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/applicationPorts` (applicationPorts .Spec.Containers) ]]”
    [[- end ]]
    env:
    – name: POD_NAME
    valueFrom:
    fieldRef:
    fieldPath: metadata.name
    – name: POD_NAMESPACE
    valueFrom:
    fieldRef:
    fieldPath: metadata.namespace
    – name: INSTANCE_IP
    valueFrom:
    fieldRef:
    fieldPath: status.podIP
    – name: ISTIO_META_POD_NAME
    valueFrom:
    fieldRef:
    fieldPath: metadata.name
    – name: ISTIO_META_CONFIG_NAMESPACE
    valueFrom:
    fieldRef:
    fieldPath: metadata.namespace
    – name: ISTIO_META_INTERCEPTION_MODE
    value: [[ or (index .ObjectMeta.Annotations “sidecar.istio.io/interceptionMode”) .ProxyConfig.InterceptionMode.String ]]
    [[ if .ObjectMeta.Annotations ]]
    – name: ISTIO_METAJSON_ANNOTATIONS
    value: |
    [[ toJSON .ObjectMeta.Annotations ]]
    [[ end ]]
    [[ if .ObjectMeta.Labels ]]
    – name: ISTIO_METAJSON_LABELS
    value: |
    [[ toJSON .ObjectMeta.Labels ]]
    [[ end ]]
    [[- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) ]]
    – name: ISTIO_BOOTSTRAP_OVERRIDE
    value: “/etc/istio/custom-bootstrap/custom_bootstrap.json”
    [[- end ]]
    imagePullPolicy: IfNotPresent
    [[ if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port` 15020 ) “0”) ]]
    readinessProbe:
    httpGet:
    path: /healthz/ready
    port: [[ annotation .ObjectMeta `status.sidecar.istio.io/port` 15020 ]]
    initialDelaySeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds` 1 ]]
    periodSeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds` 2 ]]
    failureThreshold: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold` 30 ]]
    [[ end -]]securityContext:
    privileged: false
    readOnlyRootFilesystem: true
    [[ if eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) “TPROXY” -]]
    capabilities:
    add:
    – NET_ADMIN
    runAsGroup: 1337
    [[ else -]]

    runAsUser: 1337
    [[- end ]]
    resources:
    [[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -]]
    requests:
    cpu: “[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` ]]”
    memory: “[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` ]]”
    [[ else -]]
    limits:
    cpu: 500m
    memory: 128Mi
    requests:
    cpu: 100m
    memory: 128Mi

    [[ end -]]
    volumeMounts:
    [[- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) ]]
    – mountPath: /etc/istio/custom-bootstrap
    name: custom-bootstrap-volume
    [[- end ]]
    – mountPath: /etc/istio/proxy
    name: istio-envoy
    – mountPath: /etc/certs/
    name: istio-certs
    readOnly: true
    [[- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount` ]]
    [[ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount`) ]]
    – name: “[[ $index ]]”
    [[ toYaml $value | indent 4 ]]
    [[ end ]]
    [[- end ]]
    volumes:
    [[- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) ]]
    – name: custom-bootstrap-volume
    configMap:
    name: [[ annotation .ObjectMeta `sidecar.istio.io/bootstrapOverride` “ ]]
    [[- end ]]
    – emptyDir:
    medium: Memory
    name: istio-envoy
    – name: istio-certs
    secret:
    optional: true
    [[ if eq .Spec.ServiceAccountName “” -]]
    secretName: istio.default
    [[ else -]]
    secretName: [[ printf “istio.%s” .Spec.ServiceAccountName ]]
    [[ end -]]
    [[- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolume` ]]
    [[ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolume`) ]]
    – name: “[[ $index ]]”
    [[ toYaml $value | indent 2 ]]
    [[ end ]]
    [[ end ]]
    Error: failed to start patch cert loop mutatingwebhookconfigurations.admissionregistration.k8s.io “istio-sidecar-injector” not found

    Any solution?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.