Four-time Olympic champion Hoy had to settle for silver in his favoured Keirin event, having won the three previous World Championships he entered, in 2007, 2008 and 2010.
Pendleton's bid for a fifth consecutive world sprint title ended at the semi-final stage as her perennial rival Anna Meares of Australia won 2-1 en route to claiming the gold medal.
The results meant Britain have one gold, three silver and four bronze medals from eight Olympic events, with Dani King adding a bronze in the non-Olympic scratch race tonight.
Although disappointed, Hoy was unfazed and believes he may derive new motivation from having lost the rainbow jersey - something which can also be attributed to Pendleton and the team as a whole.
He said: "History has shown for me that I come back from defeats with extra motivation.
"I'll be using tonight and this week to spur me on through the off-season and just to work as hard as I can.
"Even if I'd won the gold medal tonight you can't sit back, you're always looking for ways to improve - that's what's kept us near the top.
"We still have events we're confident of winning gold medals in."
Hoy's second place in the Keirin meant the Scot completed the penultimate World Championships before the London 2012 Olympic Games with a haul of one silver and two bronze medals from three events.
Hoy said: "I'm disappointed not to win the gold tonight, it would have been a lovely way to finish the week off.
"But I think a silver medal from a very hard-fought final is nothing to be ashamed of.
"Three medals out of three events and eight from eight Olympic events... we're not far away from where we want to be.
"I'm proud of my performances this week. I know I'm not far away from it and when I'm at my best it's going to take a pretty special performance to beat me in London."
The 35-year-old had a stunning record in the event which begins behind a motor-paced Derny bike.
He won the world title in Palma in 2007, Manchester in 2008, Olympic gold in Beijing and reclaimed the world champion's rainbow jersey in Copenhagen in 2010 having missed the 2009 event in Pruszkow due to injury.
Hoy was boxed in at the start of the final lap but powered through the field.
However, Australia's Shane Perkins held on to take gold and Hoy had to settle for silver, with Holland's Teun Mulder taking bronze and Britain's Matt Crampton finishing fourth.
Hoy added: "I didn't have much in my legs tonight but I put everything I had into it and just came up half a wheel short."
Olympic champion Pendleton relinquished her hold on the sprint crown she had held since 2007, picking up bronze.
She said: "I knew coming into this competition that I wasn't in the form of my life; I'm committed to a two-year programme.
"I'm really pleased to come away with a medal. I haven't missed a semi-final since 2003, which I don't think is a bad run - you wouldn't expect that from a football or a cricket team, would you?"
Beijing silver medal winner Meares had the edge in their bout, taking the best-of-three semi-final 2-1.
Meares went on to take her first world sprint title with a 2-0 final win over Simona Krupeckaite of Lithuania.
Pendleton responded to take bronze with a 2-0 win over Olga Panarina of Belarus and will ride in the Keirin on Saturday.
Pendleton said: "I was the only person to take a ride off her (Meares) in the competition, which just goes to show that even if she's in the form of her life I can still get her."
Pendleton said prior to Apeldoorn she was unsure whether she wanted to be in possession of the rainbow jersey in London, with the additional expectation of being world champion.
The 30-year-old from Stotfold, Bedfordshire reiterated that statement.
She said: "Wearing the stripes is wonderful, but wearing them is sometimes a burden, you do feel like you're wearing a target.
"To have somebody else wearing them and to have someone to chase is a really positive thing."
Sam Harrison finished 12th in the six-event omnium, his best result a third place in the one-kilometre time-trial which completed the event.
King completed the day's racing with bronze in the 10-kilometre scratch race behind Holland's Marianne Vos.
Laura Trott, who won team pursuit gold alongside King and Wendy Houvenaghel on Thursday, was 18th halfway through the women's omnium.
