DevOps – Development and IT in the Era of Agile

Posted on Aug 17, 2021

Looking at job boards and org structures these days one will often come across the term DevOps. Everyone is doing it but there is a great deal of confusion about what it means.

For many years we knew very well the role of the IT department and of its important responsibilities. On the other side of the “fence”, we had software development teams.

Time of Agile

Software development went through drastic changes in recent years – waterfall, XP, and now Agile. The main goal for these changes was to improve the effectiveness of the software development teams. It involved the proactive discovery of requirements and developing solutions through the collaboration of self-organizing, cross-functional teams and being closer to customers and end-users. Practice includes adaptive planning, incremental development, early delivery, and continual improvement. Agile encourages quick reaction to changes in requirements and resource availability.

Development has been already absorbed and incorporated the Quality Assurance team. Code delivered must be stable and a rapid development cycle requires close collaboration between two teams. We are going through a new “merger” now.

The software does not exist in a vacuum. It depends on infrastructure both for development in-house and in production. Here is where the IT team becomes essential in supporting dev activities or becomes a roadblock that slows momentum.

Practice shows that the best operating and effective team must blur boundaries between Development, Quality Assurance, and IT. This is where the DevOps term is born.

IT is not DevOps, neither DevOps is IT. As we will discuss below, DevOps is a robust process that incorporates an entire stack of teams in the software company and elevates it to the next level.

Theory

IBM defines DevOps process as “a software development process and an organizational culture shift that speeds the delivery of higher quality software by automating and integrating the efforts of development and IT operations teams – two groups that traditionally practiced separately from each other, or in silo”. It does not stops there and must extend across other essential teams – Quality Assurance, security, compliance, product management and more.

DevOps is a compound term of Development (Dev) and Operations (Ops). It is not the siloed team responsible for specific “it”, but the versatile aggregate of various roles in your organization allowing bring everyone together with one goal – the ability to better respond to customer needs, streamline the development process, and achieve business goals faster. The end goal is to build better products faster without jeopardizing customer satisfaction.

It is essential to understand that you cannot simply rename the IT team as DevOps, nor can you make the dev team be responsible for DevOps activities. As with the Agile process in development, there must be a mental shift from linear approach (transition from Waterfall to Agile in software development) in the IT department. For the dev team to move fast, the IT team must have buy-in and be able to move fast as well.

There are few considerations that must not be forgotten or overlooked – IT teams traditionally are responsible for IT infrastructure security and compliance. This is a very regulated and tedious area that requires careful orchestration and practices to ensure the stability and security of the software and client’s data. Said that it is not prohibitive to structure IT activities to incorporate agility. If anything, it is easier, in my opinion, for IT to be agile in the era of cloud computing and infrastructure than it is for some software engineering teams.

Continuous Integration and Continuous Delivery become cornerstones of any successful development team build. It ensures that repetitive tasks are automated – from continuous automated testing to the ability to certify and deploy new versions.

Corner Stones of DevOps

The same IBM white paper defines the following steps integral for any development:

  • Planning – identify and scope new features for the current dev cycle based on company’s product goals and client’s feedback. Rapid iterations allow for some flexibility and “catch up” approach in requirements as next cycle will bring more feedback and more clarity of what need to happen.
  • Development – each team can implement their own steps of getting from A to B in implementation and choose between varieties of approaches – test driven development, behavior-driven development, contract programming, etc.
    Main thing style must accommodate ability fast deliver quality code in each iteration.
  • Integration – CI or Continuous Integration – You cannot ensure rapid development without clearly defined and automated way to test and ensure that software is not broken in each iteration. Yes, you can beef up your QA team to allow for full regression testing in reasonable time, but to ensure that testing fatigue would not set in, automation is a key. It is a goal for QA team to spend reasonable time on scripting test where and as deep as possible so unit and UX testing can be plugged into Build process.
  • Deployment – CD or Continuous Deployment – all above steps are meaningless if software cannot be deployed properly ensuring all setting and prerequisites are met. Deployment downtime only can be limited if and when there is reasonable amount of automation in deployment and deployment validation. While humans are perfect and we can do many things over many cycles, assurance that all deployment steps are accounted for is possible only through automation. If you have to provide SOC compliance evidence, this step becomes even more important and evidence collection is automated and repeatable.
  • Operations – After production delivery development team’s role could phase out. Performance and Security monitoring, along with audit, and client support becomes routine day-to-day activity. It is important to mention that in agile cycle developers must be very close to client’s pain, be able to live and feel it every day. Only that way software will evolve in organic way. This leads us to …
  • Learning – CF or Continuous Feedback – teams must talk. You must demolish walls, you must have buy-in from all teams involved. It is essential to ensure that there is no “broken phone line” between clients, development, IT and QA teams. Exchanging information in most effective way is crucial. While learning from client’s feedback and any discovered errors we are going back to step 1 – Planning – above. New cycle starts.

Agility of Agile Process

It is important to recognize that the Agile process is not formal to the same degree as the Waterfall models of the past. Your team must be flexible and adopt practices best suited for your product, your organization structure, and your budget. This is the beauty of the Agile approach – you can scale up, scale down, expand and shrink your iterations to gain as much as possible from it – be flexible.

Agile Process and Tools

None of the above is possible without proper tools which will help to streamline and automate DevOps process adoption. Any team needs all or most of the tools to be able to automate steps in the above list – Source control tools, test and build automation tools, team management and communication, workflow and visualization.
I am not going to spend much time here discussing available options because tomorrow there will be new and better tools as the software stack constantly changing. Once you decide to adopt the Agile approach, please spend proper time deciding on tools that will fit your budget and your requirements, but do not overlook the importance of tools that will make your Agile and DevOps journey easier. Looks at GCP, AWS, and Azure offerings if you are in the cloud, or even if you are not. But do not stop there, there is plenty of good open-source and proprietary tools on the market and with each day new products would appear. The main thing to remember is to ensure that what tools you would settle on should not restrict your ability to continue to evolve.

More Variances

As DevOps methodology evolves a new variances start to appear. One of the recent extensions represents growing pain associated with security – DevSecOps – in addition to focusing on the collaboration of Development and Operations teams, the Security team is folded in to ensure that software is designed and proactively addresses growing concerns associated with data breaches.

Who is DevOps Engineer

While DevOps is more of a business philosophy than anything else, DevOps Engineer is still a real thing in the job market today. Companies would target these positions to cover the necessity to program for automation and monitoring tools. If in the past bash script knowledge was enough, with the complexity of cloud environments and integrated systems, the person in this role becomes closer and closer aligned with the software team than with IT in classical terms. Not only knowledge of CI/CD tools is required, but knowledge of the cloud stack, various APIs, and things like Lambda, Python, Java, Ruby, Git, Jira, and even languages and IDEs used by supported dev teams. The boundary between IT and dev is further blurred to the point where there would be very little distinction between knowledge required.

Transformation is complete

When adopting DevOps methodology software organizations must be ready and accepting of the fact that everyone is working toward one goal, there are fewer and fewer inter-team boundaries and the distinction between IT and development disappears. For the organization to be effective in today’s world we must now focus more on clear communications, better requirements, and eliminating “org silos”.

Posted in Product Management, Technology | Tagged , , , | Leave a comment

Database as a Service, High Availability, Horizontal Scalability, and Multi-master Architecture

Posted on Jul 14, 2021

DISCLAIMER: This post intended to aggregate research performed while evaluating various DB solution fitting specific requirements. While it may be a bit lengthy, I hope it would give the reader good starting point while deciding which path to take.

SkySQL was reviewed in more details, and this article presents in-depth review of “last-mile” considerations. The intent is not to force reader into particular solution, but to share all accumulated information.

Introduction

While deciding about changes to the database layer at the core of your system many questions and concerns are being raised.

  • Will my system perform as expected after migration?
  • How much effort is required to migrate existing solution?
  • Will it require serious code refactoring?
  • How a new DB solution will fit existing CI/CD practices?
  • What would be impact on product budget in the near future and a long term?
  • Is it available on current cloud platform? Can it be cloud provider agnostic?
  • Can my DevOps team support it and/or will require extensive additional training?

Team needs to take a few things into considerations to ensure a smooth transition and minimize downtime while gaining as much from the migration.

Project’s OKRs

Let’s establish more narrow goals and assess our milestones:

  • Evaluate a Database-as-a-Service vs database as an appliance in the cloud.
  • Ensure High Availability for the database layer.
  • Add Horizontal Scalability and Multi-Primary support.
  • Minimize impact on existing application while minimizing adjustments to the code (in-place drop-in is ideal, i.e. keep standard connectivity to database through non-API DB access layer, if possible).
  • Ability to integrate into existing CI/CD workload.

For purpose of our specific research a few additional requirements also apply:

  • Existing solution uses MariaDB database engine with Primary and Replicas in place.
  • Downtime should be no more than few hours.

Let’s review each requirement, its benefits, and then options available with MariaDB engine.

Database-as-a-Service

“Database-as-a-Service (DBaaS) is one of the fastest-growing cloud markets. According to Gartner, 75% of all databases will be running in the cloud by 2022. The first relational DBaaS offerings were created by cloud service providers – Amazon RDS, Microsoft Azure Database and Google Cloud SQL. These services started with MariaDB, MySQL and/or PostgreSQL.”

Benefits

  • Save money by combining hardware, hosting, software and support cost into one package.
  • Rapid Provisioning of DB instances takes shorter time due to automation and streamlining of the associated activities.
  • Outsourcing of administration and monitoring reduces on-premise IT team activity requirements and optimizes time to market and results in further cost savings.
  • Enhanced Security created through “by-design” security practices helping with compliance and security breaches. Off-site hosting lowers complexity of on-site security requirements.
  • Improved monitoring and tracking of time, space and resource consumption.
  • Server Space requirements are reduced and optimized.
  • Scalability is achieved through advanced provisioning via provider’s offering.

High availability / Always-On Architecture

“High availability (HA) is a characteristic of a system which aims to ensure an agreed level of operational performance, usually uptime, for a higher-than-normal period.”

To facilitate High Availability and Always-On Architecture number of considerations need to be taken.

MariaDB already provides several options (reference):

  1. MariaDB Replication – MaxScale’s MariaDB Monitor (mariadbmon) detects primary server failure and promotes the most up-to-date replica based on Global Transaction ID (GTID), waits for that replica to execute any transactions in its relay log, and begins routing queries to it.
  2. MaxScale’s Galera Monitor (galeramon) is used to minimize application impact upon server failure. Additionally, MaxScale may be used to assign primary and replica roles to database instances within a cluster to support read/write traffic splitting and to minimize the risk of certification failures.
  3. Multi-Node Enterprise ColumnStore – MaxScale’s MariaDB Monitor (mariadbmon) determines which node is the primary server, and automatically performs fail-over upon primary server failure.
  4. MariaDB Xpand – Xpand divides tables and indexes into slices, and it distributes those slices among all the nodes. Xpand maintains replicas of each slice, so that it can recover from a node failure without loss of data.

The replication approach is the most common one today and works well with non-distributed processing.  However, it puts certain limitations on how architecture can scale horizontally.

Multi Master Architecture / Horizontal Scalability / Continuous Availability

Horizontal scalability: The ability to increase capacity of the server by connecting multiple hardware or software entities so that they work as a single logical unit. HS can be achieved with the help of clustering, distributed file system, and load balancing.

Continuous availability: The infrastructure (or the applications running on it) cannot be interrupted at all. Essentially, there is no allowance for any outage, either planned or unplanned.

This is where the multi-master approach (ex: MaxScale’s Galera Cluster – option #2 above) comes in.

“Galera Cluster turns the popular MySQL and MariaDB server into a multi-master replication enabled solution with an automated member provisioning and joining process and no-data-loss guarantee with a proper configuration”.

With the maturity of MariaDB Xpand Distributed SQL architecture, MaxScale can be substituted to achieve even greater MMA flexibility.

The concept of MMA is not new and is used with distributed AD controllers, multi-zone data centers, etc.

Here is an old introduction video that provides some top-level details about how Galera architecture extends classical Replication approach – Galera Cluster for MySQL introduction – YouTube

Official information on Galera can be found here – https://mariadb.com/kb/en/what-is-mariadb-galera-cluster/

All MariaDB Galera features listed below are now rolled into Enterprise cluster setup and are supported natively.

Let’s review 3 workflows:

Node Failover

MMR can be used to protect the availability of a mission-critical database. For example, a multi-master replication environment can replicate data in your database to establish a fail-over site should the primary site become unavailable due to system or network outages. Such a fail-over site can also serve as a fully functional database to support application access when the primary site is concurrently operational.

The database can be configured for automatic fail-over, which enables the main node to fail over to a different master site if the first master site fails.

Load Balancing and Multi-Master Replication

MMR is useful for transaction processing applications that require multiple points of access to database information with the following goals:

  • Distributing a heavy application load.
  • Ensuring continuous availability.
  • Providing support for edge computing efficiency.
  • Protect from geographical impacts.

The main goal of MMR is to ensure that data is available when and where you need it. The following describes several different scenarios that have different information delivery requirements.

Applications that have application load distribution requirements commonly include customer service-oriented applications.

What is MaxScale

MariaDB MaxScale is a “DB conductor” which enables database access orchestration and automatic load balancing and has the following characteristics:

  • High availability, scalability, and security.
  • Simplifying application development by providing unified direct client connections, native MariaDB look & feel.
  • Transparent load balancer with support of “Write to primary, read through replicas” distribution.
  • Automatic membership control, and primary recovery on primary loss.

Other Considerations

When comparing various DB options, it is important to note that there could be simply a difference in DB engines that will drive your decision. As an example, while MySQL and MariaDB are close in theory, and MariaDB supports MySQL backward compatibility mode, engines are different and software must be tested to ensure a smooth transition between the two. Furthermore, after version 10.2 MariaDB is no longer considered a drop-in replacement for MySQL.

DBaaS Solutions

We have established the foundation, let’s look at what is offered in a cloud. There are more options available, but we keep the list short for the purposes of this research.

  • Cloud SQL – mySQL / SQL Server / PostgeSQL – hosted in GCP by Google team. HA* options. No multi-master support. No Maria DB support.
  • SkySQL MariaDB – hosted in GCP/AWS by MariaDB managed services. HA and MMS/MMR.
  • AWS Aurora – mySQL / PostgeSQL – hosted in AWS by AWS team. HA and MMS.
  • Azure Database for MariaDB/MySQL – MariaDB/MySQL – hosted in AWS by AWS team. HA enabled. Scalable.
  • Azure Cosmos – fully managed NoSQL DB – hosted in Azure. HA and MMS.
  • Google Cloud Spanner – non-traditional RDMS as data access requires API abstraction.  HA enabled.

MariaDB/MySQL based DBaaS Solutions Side-by-Side

Cloud SQL Features

Cloud SQL second generation is announced on Aug 1, 2016 for general availability and introduces some changes not applicable in the previous iteration.

  • Fully managed with 99.95% availability
  • Integrated
  • Based on MySQL 8.0.18 / 5.7 – slightly behind from official but not terribly (8.0.23 is officially available). As of time of this research, CloudSQL no longer includes information about current build used in official release notes.
  • Reliable – SSAE 16, ISO 27001, and PCI DSS compliant and supports HIPAA compliance
  • Offered by Google, but is lesser offering than SkySQL or Cloud Spanner.
  • No Memcache support
  • Read replicas are supported, but not as clustered solution. Application must maintain direct connection to a read replica and perform load balancing internally.
  • CloudSQL High Availability configuration
    Note: The HA standby instance cannot be used for read queries. Read replicas cannot have HA mode enabled.
    HA is achieved by disk replication between Zones to a standby instance. In order for standby instance be brought online, engine on standby would be brought online and would process any pending binlog transactions which may take up to 3 minutes.

HA in both Amazon RDS and GCP CloudSQL implemented via Shared Storage topology (DISK level synchronous replication across different zones). Actually, on the failure of the Primary server watchdog will switch to the instance which is in the standby state. During boot, MariaDB/MySQL service will analyze binary logs and roll back all uncommitted transactions. So, that 2-3 minutes are the time required for boot and applying binary log to data. In my opinion, 2-3 minutes is a very optimistic time. Probably GCP is using Shared Storage topology because it is much cheaper (MaxScale analog – MySQL Router is available under a dual license, probably this obstacle don’t let to integrate it as a solution for cloud providers).

HA in SkySQL implemented via MaxScale. So it is not correct to compare CloudSQL and SkySQL side-by-side. Also additional MaxScale layer means that if it fails no matter how many replicas are in place, DB will stay unavailable until MaxScale instance is recovered. SkySQL now offers HA option for MaxScale now to address the issue.

SkySQL Features and Benefits

SkySQL is MariaDB’s answer to CloudSQL and builds on top of extensive enterprise knowledge the MariaDB team acquired over the years.

SkySQL is managed solution which expands MariaDB, MaxScale, and Galera Cluster offerings to the cloud and includes all features from individual products. It is available in GCP and AWS allowing for easy hosting migration.

The solution is managed by the SkySQL team – the professional services arm of MariaDB Corporation.

SkySQL Features

First announced March 31, 2020. GA announced May 12.

Current versions:

  • MariaDB Enterprise Server – 10.4.14-8
  • MaxScale – 2.5.3
  • ColumnStore – 5.4.1
  • Xpand – used in Distributed SQL configuration. Would resemble CloudSpanner in behaviour without API overhead
  • Galera 4.0+ – offered as an option but can be overlooked by proper tuning of MaxScale cluster or via Distributed SQL configuration.

Other features:

  • Fully managed with 99.95% – 99.995% availability
  • Cluster Nodes split across zones within the same region with DR nodes in different region if necessary.
  • TLS 1.2/1.3 required for all connections

Please refer to the attached October 2020 SkySQL Announcement for additional details.

SkySQL usage is reviewed below in Transactional usage and offered in 4 configurations:

  • Transactional Standalone – 1 node
  • HA – primary and 2 replicas – 3 nodes
  • Galera – 3 nodes

SkySQL Benefits

  • Always up to date: SkySQL is kept current and updated to stay in line with current products available
  • Cloud Distributed: SkySQL includes a fully distributed cloud data warehouse that now provides massively parallel processing (MPP) for scalability and high availability on large datasets.
  • End-to-end security: All SkySQL databases are secure by default, built from the ground up to provide the ultimate security in the cloud. SkySQL now also enforces secure SSL/TLS connections for any database access, avoiding exposure of data due to insecure defaults or configuration choices. AD integration is available on request.
  • Disaster Recovery: SkySQL combines nightly backups, persistent disks, point-in-time rollback and cross-region replication to minimize the recovery point and recovery time objectives (RPO/RTO) of mission-critical, production databases.
  • Expanded monitoring: SkySQL monitoring shows the status and all vital metrics for database instances and is highly customizable. Health state of the nodes within defined scalability strategies.
  • Radically reduced complexity for application development: SkySQL provides a single connection point for applications rather than exposing individual database instances, primaries or high availability replicas. SkySQL manages read/write-splitting, seamless fail-over and application session migration. The result is fault tolerance and efficient use of resources that is completely transparent to developers and end users.

SkySQL for Distributed SQL

Alternative to SkySQL cluster and Galera is SkySQL for Distributed SQL.

  • Based on Xpand technology.
  • Requires min 3 nodes.
  • Single node failure without loss of data. Auto healing without intervention.
  • Elastic scale-up/scale-down infrastructure topology.

SkySQL SLA Considerations

Official Service Downtime metrics and Service Credits

  • 99.95% service availability in multi node configuration – ex: 22 min max over 30 days
  • if exceeded 99.95% Service Credits will be applied per calculated monthly uptime percentage prorate.
  • Excludes impact of GCP and GKE engine.

SkySQL Support Options

As presented in the Pricing section several tiers are available:

  • Normal response time 4-8 hours for Standard tier.
  • SkyDBA / Premium – in minutes / self managed / self driven – additional cost.
  • DBA backup recovery – normal – severity 1 – 8 min avg, 30 min SLA – immediate action upon reporting / detection.
  • GCP Kybernetes engine takes on recovery of the lost node and will rejoin the cluster – does not require ticket.
  • Patches are applied every 3 months with 2 weeks advanced notice with ability to delay.
  • Patches are first applied to replicas. Time can be agreed on.
  • Testing instance can be spin off for advance evaluation.
  • 7 days daily backup option is included. Backup retention policy can be extended for additional pass-through cost of GCS storage. Nearline storage class is used.
  • AD authentication for access can be implemented during on-boarding.

SkySQL vs CloudSQL

SkySQL vs CloudSQL comparison provided by (by SkySQL team)

Light recap – read above for a more detailed comparison.

  • CloudSQL is mySQL based vs MariaDB in SkySQL
  • CloudSQL caps write throughput forcing to upgrade instance type.
  • CloudSQL only allows Read replicas
  • In addition to Primary/replica supported by CloudSQL, SkySQL offers Multi-master and Distributed SQL configuration based on MariaDB Enterprise Server nodes for transactional access.
  • CloudSQL HA option is based on Hot/Spare two node configuration. While “Hot” node is alive, all traffic routed to it. Spare is used for auto healing when main node fails. Per documentation failover can take up 2 minutes to complete transition. May require transaction replay if caught in-flight.
    SkySQL failover takes less than a second. No transaction replay if caught in-flight required.
  • CloudSQL does not offer automatic load balancing.

Expect things to be slightly slower

While analyzing stats from your staging setup before making a final decision, do not overlook the fact that you are looking at Apples and Oranges. While the DBaaS engine may be a drop-in for your project and not many physical changes are required to keep it running, one thing is important – you are changing IT architecture. Even with GCP dedicated interconnect, you are still looking at virtualized environments and new considerations need to be made for network pipes length, edge networks, regions, and zones. Don’t be dissuaded, by making a few important, and may be long postponed, changes to your current topology and data patterns, you should be able to regain “lost” speed.

Said that, after careful review and tuning comparable speeds were achieved. Adding to it ability distribute processing load between nodes, final result should be satisfactory.

Conclusion

This article makes attempt to give the reader enough information and references to make the best choice suitable for your application configuration while satisfying your CI/CD requirements and I hope you find it useful. Google Cloud, Azure, and AWS offer many different database options and it is important to choose a DBaaS solution that makes your project a success. Good luck.

Posted in Product Management, Technology | Tagged , , , , , , , , , | Leave a comment

Death of CentOS? NOT… Think “rebirth” instead

Posted on Mar 31, 2021

I am active user of CentOS in production for many years and counting. While Windows was and still is my OS of choice on desktop, actively using WordPress and LAMP stack in general lead me to CentOS as a base platform to host our web applications and solutions. We are now equally using Debian in our deployments.

When RedHat announced shift of focus from CentOS Linux to CentOS Stream in December of last year it caused a stir in Linux/CentOS community. It was still acknowledged that announcement was botched and misleading. For me bells started to ring when IBM first acquired RedHat. Usually such changes lead to more “enterprisy” approach. And year later we now see it happening. How can we generate more $$$ for our shareholders has its meaning and drives future decisions? Yes, it is unclear what driven it and we would be told stories what lead to the decision, but there is old saying – follow the money – which most of the time is true.

Problem with CentOS was that it was sitting “downstream” from RHEL and led to the notion of “why pay for RedHat Unix when you can get even more stable CentOS for free”. Historically CentOS would receive changes even later than they are in RHEL/RedHat. I think thinking behind RedHat’s decision to move things around were rooted from the notion that CentOS cannibalizes on RedHat corporate sales.

New approach is that CentOS Stream, which is replacing CentOS Linux, is now moved “upstream” getting changes before RHEL gets them officially. This lead to a scare in the community and triggered people considering abandon then platform, i.e. “death of CentOS”.

“Reports of my death have been greatly exaggerated”, replied Mark Twain

or “The report of my death was an exaggeration” to be more accurate when asked to comment on rumor that he was dead.

CentOS is not dead. It changed its model. CentOS Project, it’s governing board and SIGs are still committed to CentOS growth.

Benjamin Porter wrote great detailed analysis of the story here – CentOS is NOT dead. Please Stop Saying It Is (at least until you read this). Please read it and don’t overlook links to internal blogs and announcements in the article, it worth the time spent reading ten fold.

Off course there is Debian and other Linux distros. Everyone is free to make their choices, just don’t base your decisions on misleading statements and news especially driven by enterprise marketing teams like these from RedHat FAQ.

If you’re using CentOS Linux in a commercial deployment, we suggest you look at moving to RHEL for the added management technologies, security, and support that are an integral part of the RHEL subscription. Our sales teams can help you identify the appropriate offerings that match your use case

Good luck and may the force of Open Source be with you always.

PS: For more information on CentOS refer to CentOS Project Home
Posted in Piwigo, Technology | Tagged , , , , | Leave a comment

GreyDragon v.1.3.5 has been released

Posted on Mar 4, 2021
  • Verified compatibility with PIWIGO 11.x
  • Translation files synced from repo
  • Fix misalignment of admin section header for PW 11+
  • Fix issue with plugin detection

Posted in Piwigo, Technology, Web-design | Tagged , | Leave a comment

Making your network connected HP ENVY Photo 7855 print again

Posted on Feb 19, 2021

HP printers seems to always being stop-and-go solution. Anything from upgrading to new version of Windows, switching local to network or Wi-Fi can throw them off. Over years I had number of HP and Canon printers and latest device is HP ENVY Photo 7855.

It started all great – new printer arrived in good condition. Installation went well, Windows 10 recognized it and Store has proper apps to use. It appeared we have had a winner.

Note: Printer was connected over Wi-Fi with dedicated IP provisioned on secured locked down network.

After a while I started to notice that printer will not always register properly on restart/wake up and would become unresponsive to print requests. As a scanner it would continue operate without issues and scan as PDF to computer with ease.

Firmware, driver updates, software refresh did not address the issue that sending documents to be printed ended with generic error that printer unable to print. HP Smart was still recognizing printer presence. I was able to print status page, clean print heads, any other maintenance operations just fine, but trying to print from browser, PDF or other apps end up with failure. HP Print and Scan Doctor utility would consistently “fix” port issue.

This was a clue to follow…

When you dig in printer’s settings you will notice WSD port.

WSD is not a port but a ‘port monitor’. WSD devices communicate using a series of SOAP (Simple Object Access Protocol) messages over UDP and HTTPs BUT NOT local IP addresses. Therefore, there is not an IP address for WSD devices.

To find port associated with printer in Windows 10, open the Start menu by clicking the Windows icon on your taskbar. Start typing Control Panel until system finds it in apps and click to open. Select View devices and printers. Select your printer in Printers section and notice that Print server properties is activated in top bar – click to open.

Now the magic…

Scroll through the list of ports. You will see infamous WSD port – ignore it. Look for any ports marked as Standard TCP/IP Port. Check that none of them associated with your printer IP.

Printer IP can be found in your printer’s LED menu for your printer’s network connection. Printer must be setup to use Static IP. If you tap on network icon and drag it down, Info section would be opened and just below printer’s name, you will find printer’s IP Address. Write it down.

Go back to Print Server Properties and Add Port. In next step select Standard TCP/IP Port, click New Port and follow Wizard steps – provide you printer’s IP and Port Name (same as IP), complete other wizard steps. Once finished, a new port will be added into the list.

If port for our IP already existed, click Configure Port and verify that Printer Name or IP Address is correct, Raw protocol is being used, Raw port number is normally set to 9100. Click OK to exit Print Server Properties and save all your changes.

Almost done… but one more step – now we need to tell Windows which port to use.

In Devices and Printers dialog right click on your printer’s icon and select Printer Properties. Navigate to Ports… again. You will notice that port’s list now has check boxes next to ports available and WSD Port is selected. We need to fix that – find your new, just off the press Standard TCP/IP Port. Select it and make sure check box is ticked for it. Click Apply to save your great work.

Note: This last step most likely would need to be repeated every time a new driver update is installed as HP will revert your choices back and make printer inoperable again.

Switch to General tab and hold your breath in excitement while enjoying your beautiful test page being printed.

I have no idea why HP Printer Setup utility or HP Smart cannot configure printer properly for static IP use. It does see the printer. It communicates with it. HP Scan and Capture utility works just fine.

I cannot fathom why HP Smart drivers end up being not so smart.

I hope this little writing of mine helped someone avoid pain trying figure out why lovely HP printer is not so loving.

Posted in Scratch Pad, Technology | Tagged , , | 1 Comment

Firefox 77 and Awesome expanding address bar

Posted on Jun 5, 2020

Awesome design decisions

I do not have much time to write these days, but this situation made enough annoyance for me to warrant this content.

First, kudos to Firefox team for well maintained browser.

Then…

In general, I am one of the stubborn one, who do not like fluidity of Chrome and its ever changing behaviour. In the past, IE and Microsoft was the one breaking standards. Now it appears that these great minds moved to Google and every new version of Chrome will give you something new and unexpected from great wisdom of all-knowing and all-seeing company. 🙂

Firefox team seems to feel the pressure and joined the party of making users happier by delivering surprising features.

In last few months one of the great things to appear was expanding/enlarging address bar.

There is probably some great reason for it to behave this way, and I respect the decision Firefox UX team made, but it is really distracting.

In Firefox 75 and 76 advanced users could use about:config settings to suppress this behaviour, but it was taken away in Firefox 77, so I will skip referencing solution there and instead focus on awesome 77+ build and we would need to do some coding.

Taking control back

Chrome is creeping in. No way to avoid its influence these days.

As with Chrome, you can define your own CSS styles for Firefox as well, if you know the tags.

First, you need to enable CSS styles overrides for Firefox.
Visiting userChrome.org will help you with step by step instruction on how to do that here.

Once you give yourself tools, next step is to start using them – override how address bar is styled.

From the same site, you can grab ready to go userChrome.css file which will return your bar under control here then drop it into chrome folder under your Firefox user profile folder. It is important that you create chrome folder if it does not exist before placing CSS there.

After making sure toolkit.legacyUserProfileCustomizations.stylesheets is set to true in your about:config and restarting the Firefox, you will get your address bar back.

PS: Additional candy is that reviewing the CSS will tell you exactly how to modify the bar in any other way you like. Fave fun, give it color, make it big to take half the screen, go crazy. Just kidding.

Posted in Technology | Leave a comment

SearchWP and XPDF support

Posted on Apr 5, 2018

Introduction

SearchWP is a nice WordPress plugin which allows to extend WordPress search capabilities to various content format beyond just page text.
Here I am focusing on PDF indexing specifically. If you have any PDF files published and want them be indexed SearchWP-XPDF extension can be deployed.

Issue

Problem is that SearchWP developers decided to rely on deployment of direct code from FooLabs.  If it is not deployed as a drop-in, functionality would be disabled.
Drop-in option is not always possible and in fact unnecessary for this plugin to function.

In distros like CentOS  PHP XPDF support is available out of the box and can be enabled by installing poppler_utils.

Solution

    • Install SearchWP Xpdf extension as you would normally do
    • In WP Dashboard warning would appear – “SearchWP Xpdf Integration requires you to download and install Xpdf…”
    • SSH to your server
    • Install poppler_utils using yum# yum install poppler_utils
    • Refresh your WP Dashboard page. Warning would not go away because code specifically looking for physical location instead of verifying XPDF support by OS
    • Now we do some hacking:
      • in your WP installation, navigate to /plugins/searchwp-xpdf/ and open searchwp-xpdf.php for edit
      • Look for “// see if Xpdf exists”
      • Comment out what follows 
        /*
// see if Xpdf exists
if ( $continue && file_exists( $this->xpdfPath ) ) {
echo 'Xpdf (pdftotext) was found, continuing<br />';
} elseif ( $continue ) {
echo 'Xpdf (pdftotext) was not found, aborting<br />';
$continue = false;
}
*/
      • Next look for function admin_notice()
      • Comment out content of the function to suppress Admin notice 
        function admin_notice() {
/*
...
<?php }
*/
}
    • Refresh WP Dashboard
    • Warning would now go away
    • Run indexing, PDF files should now be indexed properly

Footnote

I did actually reach out to SearchWP team back in May 2017 and solution was acknowledged, but almost year later plugin still relies on custom drop-ins.

Enjoy.

Posted in CentOS Maintenance, Scratch Pad | Tagged , , , , | Leave a comment